Random Stuff that Matters

Taking notes on Tiago Forte's Just-in-Time Project Management

2020-01-31T14:27:28Z

Note taking and Roam

I was heavily invested in a note-taking infrastructure during my PhD, using Dokuwiki with a bunch of my own extensions to enable things like extracting and reorganizing content by tags and my own web clipper. However, this workflow was so brittle that I couldn't even manage to transfer it easily to a new laptop, and in the years since that intense bout of literature review (early stage PhD), I've muddled through with a variety of tools, DevonThink, Google Docs, NValt etc.

Recently, I came across Roam Research, and was very excited by how they are combining some of the best ideas of wikis, refining my early idea of reorganizing content through tags, and hierarchical outliners (which I had seen, but never really paid that much attention to - turns out this plus backlinks enables really powerful things). For me, it was an interview that Tiago Forte did with the founder of Roam, Conor White-Sullivan (my notes), which "sealed the deal". I was very impressed by his vision, and have been digging into Roam and the community around it, ever since.

Since then, I've been thinking about the best ways of using Roam, inspired by some of the use cases people have shared (see some in Awesome Roam). I've tried interstitial journaling, keeping track of projects and goals, capturing interesting resources, etc.

Tiago Forte's concept of Just-in-Time Project Management

I've been following Tiago Forte for a few months now, through his prolific blog posts, newsletters, podcast appearances and tweets. He clearly embodies a "productivity first" or "learning through writing" approach, but he has a lot of useful things to say. One of the concepts I kept circling around was "intermediate deliverables" or "intermediate packages". One of the ways Roam works (possibly mirroring how our brains work) is that you can begin "circling in" on a concept without having to really define it. If you keep tagging a certain concept whenever you hear about it, once you finally visit that page, even though you've never written anything, you can see all the different contexts that you've referred to it, and in this way already get what someone called a "pointilistic portrait" of the concept.

So I've been thinking about getting back to blogging or at least sharing more informal writing publicly (this is the first blog post in three years), and also about how I could leverage better all the writing that happens anyway (in long e-mails, notes, etc). I decided that I wanted to properly understand what Tiago meant by this concept, and came across a link to his series on Just-in-Time Project Management. This is a list of about 22 blog posts, which are only for subscribers, so I paid 10$, and began reading. Since I really want to retain and organize this information, I opened one window in Roam and one window to read, and began taking notes. As I went along, I came to really enjoy the way Roam let's you collapse and expand sections, which enables you to keep an overview of all the different sections, even though it's actually many many pages, and made it easy to see how the subsequent information fit in, allowing me to go back and modify or add to previous structures etc, mirroring how I want to understand the ideas, rather than the order in which it is presented.

Recording the process of taking notes

After having processed about five blog posts, I had the idea to just turn on a screen recorder, and let it watch me work. I thought it might be fun to go back and look at my process, and perhaps others would find it fun - at least in a sped-up version. It was also helpful to keep focused, since I didn't want to have to go back and edit later, so I really resisted the urge to switch context, check email etc. I have also looked into, and regularly use, many tools for capturing content, such as Hypothes.is for web annotation, Scrivener as a "reading buffer", etc. But this time, I really enjoyed doing all of the processing synchronously. I know some people like to dump all of the text into Roam first, and then process, but I prefer doing at least a pre-selection (using annotations), and in this case, I did most of the processing synchronously. As I read, I change wordings, use bold and italic, and try to organize the notes so that it will be easy to come back to. I add lot's of links to other concepts (both ones I have pages on already, and ones I don't). I also add my own thoughts and questions, which I usually highlight in yellow to make them stand out.

When I had finished, I did one last quick pass over the whole text, re-reading to see if I could detect new connections etc. I could have done much more work to reorganize or integrate here, but I'll leave that for the next time I revisit this text. haring the process and the outcome

I've made all of my notes public (note that this is not the same database as I use privately, so many of the links which will have content in my private db will go to empty pages here), and if you are curious, you can also see the process by which these notes were made. Below is a 1000x sped up version (one minute), with links to a 100x and a real-time version.

Take-aways from Tiago's writings

Since I've already taken so detailed notes, and interleaved them with many personal observations, I will not try to summarize them here (although perhaps that would have been useful for my learning/retention). Suffice to say that I really like some of his approaches, even though I do sometimes feel like a lot of productivity advice comes from people who are self-employed as Youtubers, bloggers etc, and they naturally have a very different way of organizing their time, and evaluating "Return-on-Attention" than others.

I started out struggling with note taking as a student, first as a bachelor student primarily occupied with learning and passing exams, as well as doing just enough reading to be able to finish a paper (typically something like five sources), but also doing an Honour's Thesis with a large amount of material. Later I did an MA degree, with mostly seminar classes, and again a large research task at the end, and finally a PhD, where you are of course not only doing readings to prepare you for a momentous task several years in the future, but also thinking about publishing papers, participating in the scholarly community, etc.

(At that time, I also spent a lot of time thinking about how academics should collaborate, better formats for academic publishing, etc.) And currently, I'm employed at the Minerva Project, where I have an interesting job combining pure software development, with product work (specs, architecture, pedagogical design), and even some pure research. I still need to keep up with the literature, yet goals and criteria for evaluation are very different.

That said, I think I would have very much appreciated reading this back when I was a PhD student (and certainly finding Roam!), but it's possible that the time I spent in a tech company learning about modern management approaches also helps me better appreciate some of his writings.

PS: This blog post was authored in Roam.

From Seminar to Lecture to MOOC

2017-01-05T16:41:48Z

This summer, I successfully defended my PhD thesis "From Seminar to Lecture to MOOC: Scripting and Orchestration at Scale" at Ontario Institute Studies in Education, University of Toronto.

I have always been interested in different ways of publishing and sharing my research. With my BA thesis on community libraries in Indonesia, I had it translated to Indonesian, and also made it into an ebook.

I have an extensive page documenting my experiments with my MA thesis on Chinese Open Educational Courses. I made the thesis available in a number of formats, serialized it on my blog, had it translated into Chinese, and also published all of my raw notes (open notebook).

For my PhD thesis, I had big ambitions about having the finished thesis be a live document linking to my notes on all the papers cited (using my Researchr system), but although I wrote my initial drafts in Markdown (with Scrivener), they ended up in Word for collaborative editing and track changes.

I then wanted to create a fancy landing page, with links to resources from the thesis, multiple formats, talks, etc. But after landing in Lausanne, where I am doing a post-doc at EPFL, I never got around to it. Now it has been half a year since I defended my thesis, it is still not public at T-Space, the institutional repository in Toronto, and I decided it's time to set it loose.

I still hope to come back to add more information, formats, details etc. But for now, here are some essential links:

The thesis itself (PDF)
Poster I presented at EPFL MOOC camp
Slides from talk at EMOOCS 2017, May 22, 2017
A principled approach to the design of collaborative MOOC curricula, paper for EMOOCS 2017
Recording of talk given at UPF Barcelona on thesis, and current Orchestration Graph project, April 4, 2017
Recording of talk given in French at TECFA, University of Geneva, October 18, 2106
Recording of talk given at Knowledge Media Design Institute in Toronto, July 15, 2016
Recording of online talk given to DANCE network, July 7, 2016

Sending and receiving email with Elixir

2015-09-03T16:41:48Z

Yesterday there was a Twitter discussion about adding a default mailer to Elixir Phoenix (Storify). I've never used the Ruby on Rails mailer, but I got inspired to share my experience with both sending and receiving emails in a recent project.

This summer, I ran a Phoenix server that provided interactive content for an EdX MOOC. One peculiarity of the setup was that authentication happened through the LTI connection with EdX, so there was no need for a sign-up/login form, confirmation of email addresses etc. However, we used email for a number of other purposes.

Sending email

To send email, you need an SMTP server. While Elixir is perfectly capable of handling this on its own, I had heard a lot of stories of how email from private domains might hit spam-filters, and be difficult to configure correctly, so we looked around for an external provider. Amazon Simple Email Service is a no-frills service with a great price-point (currently 0.10$ per 1000 emails). It might not have all the bells and whistles of services like Mailgun and Sendgrid, but for our purposes it worked great.

Amazon SES has a web API, but the easiest way to use it is simply to configure it as an outgoing SMTP server. Looking around, I found the Mailman library, which was easy enough to configure (my code). You simply define a struct like this (storing credentials in a config file):

  def config do
    %Mailman.Context{
      config: %Mailman.SmtpConfig{
        username: Application.get_env(:mailer, :username),
        password: Application.get_env(:mailer, :password),
        relay: Application.get_env(:mailer, :relay),
        port: 587,
        tls: :always,
        auth: :always}
    }
  end

For each email to send, you construct a message struct like this (many examples):

  %Mailman.Email{
      subject: "#{entered} entered the collaborative workbench",
      from: "noreply@mooc.encorelab.org",
      to: [email],
      text: text,
      html: html }
  end

and then use Mailman.deliver(email, config).

To generate the email contents, I used EEx templates which I manually precompiled. You can see a list of templates, and the module that precompiles them.

Asynchronicity and error handling

This worked beautifully when I was testing it with individual emails, but when I wanted to send out a large number of emails (for example personalized weekly updates), I found that some emails were silently dropped. It turns out that Amazon SES has a daily limit, which for me was lifted to 50,000 emails (more than I would ever need), but also a rate-limitation, which for me was 14 emails per second. Apparently Elixir is just too fast, and Amazon would just return an error whenever it went above that rate.

To fix this, the first issue was to correctly detect errors, but Mailman seemed to return :ok tuple no matter what. Looking into their code, I found that requests to deliver led to some processing of the structs, and then passing them on to :gen_smtp_client.send_blocking using Task.async, and then immediately returning an :ok tuple. Personally, I think that libraries should let users decide on their own concurrency strategy, and hiding the errors is a big problem. I have a pull request pending which changes the function to call the :gen_smtp_client function directly, and in the meantime I am using my own fork of Mailman.

With this change, we do indeed get access to the error messages from Amazon, but we still need to handle catching errors, retrying etc. To solve this problem, I began writing my own task queue library, with the idea that it should be possible to specify different "Quality of Service" levels, including number of retries, max requests per second, time to wait per retry etc. My initial experiments need to be rewritten almost completely, and extracted into its own package, but even this simplified approach was used to send thousands of emails, scrape hundreds of websites every hours, etc. (1, 2)

User-specific links

As mentioned above, the application had no login screen, so we wanted to include links in the emails that let the users directly access the content linked to. I had already written a generic cache function, which stores any arbitrary Erlang term (using a custom Ecto type) returning a simple index (and checking for uniqueness). I used this, together with the hashids library to generate shorturls for specific URLs and user ids (source):

  def gen_url(id, url) do
    term = %{url: url, userid: id}
    id = Survey.Cache.store(term)
    hash = Hashids.encode(@hashid, id)
    @basename <> "/email/" <> hash
  end

When a user reached the /email/ controller, I simply extracted the id from the hashid, looked up in the cache store, retrieved the user from the database, and set the appropriate session variables to "log in" the user, before redirecting to the appropriate URL (source):

def redirect(conn, %{"hash" => hash}) do
    {:ok, [id]} = Hashids.decode(@hashid, String.strip(hash))
    struct = Survey.Cache.get(id)
    hash = (from f in Survey.User,
    where: f.id == ^struct.userid,
    select: f.hash) |> Repo.one
    conn
    |> put_session(:repo_id, struct.userid)
    |> put_session(:lti_userid, hash)
    |> ParamSession.redirect(struct.url)
  end

In each email, I also included a link to unsubscribe from all email notifications, or from the specific type of notification (for example, you might still want to receive weekly updates, but not notifications every time someone posted something), and I checked whether someone had unsubscribed before sending out notifications. These links also had the userid encoded, and just displayed a page showing "Success!", rather than a form asking the user to enter email and password, etc.

In addition to user-specific links, we could also customize the content of the emails based on user data in the database. Below is an example of an email that is both using a fancy HTML template (easy to do with EEx), and containing user-specific information:

Receiving emails

I was very pleased with how well Elixir and Mailman was able to handle the sending of emails, and I managed to use quite a lot of Elixir features in implementing the tasks above (parallelism, worker queues, genservers, Ecto and custom types, EEx templates, etc). However, just sending emails isn't in itself anything extraordinary, any framework should support it at some level. Receiving emails seems like it would add a whole other level of complexity, though.

In our project, we wanted to support small-group collaboration and communication between members who had never met, and were often separated by timezones. We already had embedded Etherpad, wiki and live chat, and we made it more likely that people would meet each other online, by sending out notifications whenever a group member entered the online environment. In addition, I had the idea of generating ad-hoc mailing lists for group discussion. However, one limitation was that we could not share users email addresses with anyone, due to privacy concerns.

I looked into whether there were some mailing list system that we could install on the server, with an API that would let us dynamically add mailing lists and members etc. However, when I came across gen_smtp (which Mailman is built on top of), I realized that Erlang could handle this all by itself - and in a very simple manner.

Since Mailman doesn't wrap the receiving functionality of gen_smtp, I had to dig into the Erlang code and documentation to understand how to set it up. While I love the flexibility you get by defining your own callback-based SMPT server, it does seem that most people simply want to receive emails and do something with the contents. Luckily, the repository has an example implementation.

I had a few hiccups figuring out how to set up the server. First, I somehow spent an unreasonable amount of time figuring out how to start the server from my Application configuration (translating the Erlang invocation into Elixir), although the result is very simple (source):

 worker(:gen_smtp_server, [Mail.SMTPServer,
        Application.get_env(:mailer, :smtp_opts)]),

where the relevant line from the config file is:

  smtp_opts: [[port: 3000, sessionoptions: [certfile: 'server.crt', keyfile: 'server.key']]]

The keyfiles are for TLS verification. The reason for putting the port number in a config file, rather than directly in the Application invocation, is that only one process can listen to the port at the same time - so if you have a production instance running, and you want to fire up an IEx to experiment with some code, they will both try to listen to the same code, and the second process will fail.

The second issue was accessing port 25 on the server, which is the standard SMTP port. I didn't want to run my BEAM process as root, but by default, only root can bind to ports below 1024. I finally found the obscure answer in a Stackoverflow post, which is that executing sudo /sbin/setcap 'cap_net_bind_service=ep' BINARY on a BINARY gives that binary the privilege of listening to ports below 1024. I ran this command against the BEAM binary, and everything worked well.

I initially just tweaked the example module in Erlang to fit my purposes, but then decided to translate the file to Elixir, while removing a lot of the documentation comments and edge cases (which were mostly there to show the possibilities). The result was a very lean module that looks a lot less scary. You can see the whole source, but I reorganized it so that the callbacks you are most likely to want to modify are at the top (shown below), and the callbacks that you should probably leave alone, are at the bottom (see source):

defmodule Mail.SMTPServer do
    require Logger
    @behaviour :gen_smtp_server_session

    def init(hostname, session_count, address, options) do
        if session_count > 40 do
            Logger.warn('SMTP server connection limit exceeded')
            {:stop, :normal, ['421 ', hostname, ' is too busy to accept mail right now']}
        else
            banner = [hostname, ' ESMTP']
            state = %{}
            {:ok, banner, state}
        end
    end

    # possibility of rejecting based on _from_ address
    def handle_MAIL(from, state) do
        {:ok, state}
    end

    # possibility of rejecting based on _to_ address
    def handle_RCPT(to, state) do
        {:ok, state}
    end

    # getting the actual mail. all the relevant stuff is in data.
    def handle_DATA(from, to, data, state) do
        Mail.Receive.receive_message(from, to, data)
        {:ok, UUID.uuid5(:dns, "mooc.encorelab.org", :default), state}
    end
    [...]

All this does is give you the option of rejecting based on from, and to addresses, and then pass the entire message to a separate function for processing.

The way we used this is to include a form in the web interface to initiate an email conversation.

Any emails sent through this form would be forwarded to all members of the group (who had not unsubscribed), with a unique from address, which encoded the group id using hashids. Replies to this email would reach the server, and be resent to all group members, without revealing the email address of the original sender.

Conclusion

So there you have it. Social notifications, personalized emails, error messages by email, URLs that automatically log you in, ad-hoc mailing lists, and all with pure Elixir and Erlang. This whole experience really made me appreciate the Erlang/Elixir ecosystem. And for sending 13,500 emails in a month, I paid less than for a cup of coffee...

It would be interesting to discuss how we can improve the tooling around emails even more. Perhaps Mailman can be extended to also cover the server aspects of gen_smtp, perhaps we need better documentation (hopefully this blog post can be a modest contribution), or better integration with other libraries. But I think we're building on a great foundation!

Email notifications about errors in Elixir

2015-08-26T16:41:48Z

This summer, I ran a Phoenix server that provided interactive content for an EdX MOOC. Given the geographical distribution of students, the server was getting hit 24/7, and I wanted a quick way to get notified about any error messages. Elixir comes with a built-in logging framework that has several levels of logging (debug, info, warn, error). Any processes crashing will emit error logs, and you can also emit them manually from your own code.

The stock logger only comes with a console backend. I ran the server through Ubuntu's upstart system with a very simple configuration script, which simply set some environment variables and paths, and then used PORT=4000 MIX_ENV=prod mix phoenix.server (I used Nginx to proxy the local port, and add SSL). I used mosh and tmux to keep a persistent connection, and tail -f to watch the upstart log. On the screenshot below, I had several panes tracking both the access log (using plug_accesslog), the Phoenix log, and also a grep on error to see the latest error messages.

Due to BEAM (the Erlang virtual machine) and OTP, Phoenix is remarkably stable. The only time the whole server went down would be because of a serious issue in the datacenter rendering the entire virtual machine inaccessible. Errors were typically bugs that would occur given a certain edge case (for example trying to render the group of a user who had not selected a group yet). They would crash the connection of the user who hit that edge case, but would not propagate up the stack. Thus it would be quite possible for an error to occur and not be noticed, since everything else would continue happily running.

I decided I'd like to get an email every time an error was triggered. Luckily, the Elixir logger was built from the start as a pluggable system, and there are already a number of possible backends, feeding into syslog, or several "monitoring as a service" systems like Honeybadger and Crashdump.IO.

I had already integrated e-mail into my system, using Mailman to send email through AWS Simple Email Service for notifications, and looking at the examples in Elixir, and the external loggers, writing a custom back-end that sends email notifications turned out to be remarkably simple.

Copying some code from the console backend, I ended up with this module, with only 50+ lines of code, most of which handles config and setup. The real work is done in these two functions, first a gen_event callback whenever a message with level error is emitted, and which simply dispatches to the log_event function:

def handle_event({:error, _gl, {_, msg, ts, md}}, state) do
  log_event(msg, ts, md, state)
  {:ok, state}
end

then the log_event function, which uses the Logger.Formatter module to get the string to mail out, briefly checks against two kinds of errors which I did not want logged by email, and then composes and sends an email, using exactly the same functionality that I use for sending a new user notification.

defp log_event(msg, ts, md, {from, to_list, format, metadata}) do
  msg = Logger.Formatter.format(format, :error, msg, ts, Dict.take(md, metadata))
        |> IO.iodata_to_binary
  if !String.contains?(inspect(msg), "GenServer :job_worker") &&
    !String.contains?(inspect(msg), "bad_charset") do
    %Mailman.Email{
      from: from,
      to: to_list,
      text: msg}
    |> Survey.Mailer.deliver
  end
end

This worked perfectly.

A trick when developing something like this, is to set the level to warn while developing. Otherwise, if you have an error in your logging code, emitting an error log message from IEx to test your code, will trigger an error log message about the error in your logger, which will trigger another one, etc. This also means that in general, it is very important to make sure that the error logging code cannot itself trigger errors. If sending the email might cause an error, it might be better for the mailer to be a separate gen_event, and for the logger to simple send a message to it.

There is no checking for duplicate errors in this system, which means that in a situation where a lot of errors are emitted, you will receive a lot of emails.

This sometimes happened when I forgot to do a mix deps.get after pushing new code that had an added dependency (I didn't use any kind of release management, but simply pushed code to Github, pulled it onto the server, and restarted the server). It once happened that an incoming e-mail triggered a panic with gen_smtp because of a failing charset - this e-mail was resent every ten minutes for 24 hours, which resulted in many error messages. It also happened that the server was rebooted into single-user mode because of a datacenter failure, which meant it could not connect to the database. Upstart will automatically restart the system on a crash, again resulting in a stream of error mails.

I'm not sure how one could handle this, because I think it makes sense to keep the error logging system as simple and robust as possible. A gen_server which handled e-mailing and kept counters would help to some extent, but not in the case of multiple restarts. In that situation, you would need an external service which aggregated emails and had conditional dispatch, and then you are getting into the level of building up a monitoring and reporting infrastructure, either internally or as a service.

However, this was a great first try - it did work very reliably, and it's also helpful to see that there is very little magic to Logger and the logger backends.

Elixir Prelude: Packing up utility functions

2015-08-21T16:41:48Z

Introduction

I've been spending the summer in China, head-deep in Elixir, writing interactive scripts for an EdX MOOC on inquiry and technology for teachers (screenshots). I will have much more to share about that project later, but now I wanted to share a small Elixir package I released on Github called Prelude (documentation). This is a collection of utility functions for Elixir that I've extracted out of my code base.

Organization

I began by scattering these utility functions all over as small defps, but when I wanted to reuse them in other modules, I began gathering them. Initially I had a single file called Prelude - modelled after the Haskell Prelude, just because it was the only file I'd ever wholesale import into my other modules.

Eventually I began separating the code out into sub-modules, which meant I went from

import Prelude

map_atomify(...)

Prelude.Map.atomify(...)

which seems cleaner.

The point of gathering "trivial" functions

Many of these functions are very simple, but yet things you use all the time - gathering them together both means less repetition and greater legibility. Compare:

x
|> Enum.map(fn {k, v} -> {v, k} end)
|> Enum.into(%{})

x
|> Prelude.Map.switch

The code in the first version is short enough that I wouldn't bother making it a separate function, but the intent of the second snippet is much clearer. I might also forget to do Enum.into(%{}), and somehow I always forget the end in anonymous functions. So, some quick time saving.

Map functions

Some of the functions are a bit more involved, mainly found in the Prelude.Map module. I was inspired by the way Clojurists often work with deeply nested maps, sometimes keeping the entire application state in one atom. This has also become relevant in Javascript-land with experiments around immutable state such as Baobab.

I see deeply nested maps much less often in Elixir. Part of it might be the architecture (we use gen_servers, message passing, and ETS tables, rather than atom-swapping), but part of it might also be because of the syntax (%{%{%{}}} is not pretty).

However, Elixir already has get_in and put_in to reach into deeply nested objects. Prelude provides deep_put, which is a parallel to mkdir -p, it will insert an object and create any part of the missing path. If you put to a path that already exists, it will turn it into an array.

Built on top of deep_put is group_by. Building the Phoenix app, I often had to extract data from the database using Ecto, and render it in templates. Let's say you have a table of students, listing their group affiliation, and which class their in. You wish to list all students, sorted by class, and then by group. With group_by, you can get the data from Ecto like this:

MyRepo.all(Student)
|> Prelude.Map.group_by([:class, :group])

The result will be a map of classes. Each class will point to a map of groups, and each group will point to a list of students in that group. Then in an EEx template, you can display the list like this:

<%= for class <- @classes %>
  <h2><%= class %></h2>
  <%= for group <- class %>
    <h3><%= group %></hr>
    <%= for student <- group %>
      <li><%= student %></li>
    <% end %>
  <% end %>
<% end %>

Wrapping up

Anyway, the code is available, so feel free to browse around. I will be using this in my future projects, and probably adding functionality that is general enough to stand on its own. I'd love to add tests, and am happy to think about naming, APIs etc, if anyone else think this is interesting.

It's also my first time to use ExDoc and Github Pages, but both were a breeze.

Fun with terminal and Vim macros

2015-04-19T16:41:48Z

I always enjoy finding new ways of automating boring tasks, and today I had a few different small tasks that I was able to solve using some terminal-fu, and Vim macros. I really enjoyed how well it came together, so I thought I'd put together a short screencast to demonstrate.

I had generated a number of discussion forum graphs for different courses. The processed data is in a json file in a directory named after the course. My task was to put an identical index.html file into each directory, create an index of the courses, with links, and modify all the landing pages to reflect the names of the courses. Not a hugely complicated task, but something you could run into day to day, and which would be quite cumbersome to do manually.

The video should be quite self-explanatory, but I've included some "show-notes" below.

As far as I know, ls doesn't have a built-in way to list only directories, so I found this snippet somewhere and added it to my .zshrc:

alias ld="ls -lht | grep '^d'"

awk is a very powerful string manipulation tool, but in this case we're just using it to get the 9th column of each line (awk '{ print $9 }')

xargs is very useful to run commands on input arguments. We need -n 1, otherwise it would put use all the arguments in the same command. With -n 1, it executes the command once for each input argument. If we didn't want to put the arguments last, we could do something like echo 1 2 3 | xargs -n 1 - J % echo % hi.

OpenEdX and LTI: Pedagogical scripts and SSO

2015-02-20T16:41:48Z

Installation
Cohorts and single-sign on
LTI to the rescue
Early experiment, group-based Etherpad selection
Confluence integration and templates
See it in action

I came up with a neat way to embed external web tools into EdX courses using an LTI interstitial, which also allows for rich pedagogical scripting and group formation. Illustrated with a brief screencast.

Last year, I experimented with implementing pedagogical scripts spanning different collaborative Web 2.0 tools, like Etherpad and Confluence wiki, in two hybrid university courses.

Coordinating collaboration and group formation in a class of 85 students was already challenging, but currently we are planning a MOOC built on the same principles. Before we begin to design the pedagogical scripts and the flow of the course, a key question was how much we could possible implement technically given the MOOC platform. Happily, it turns out that it will be easier and more seamless than I had feared.

We chose to go with EdX (University of Toronto has agreements with both EdX and Coursera), which is based on an open-source platform. Open source can mean many different things, there are platforms that are mainly developed privately in a company, and released to the public as periodic "code dumps", and where installation is very complex, and the product is unlikely to be used by anyone else. I didn't know much about the EdX code, but the fact that the platform is already used by a very impressive number of other installations, such as China's XuetangX and France's Université Numérique was already a very positive sign.

Installation

The OpenEdX code is available on GitHub, and seems to be a very active community -- more than 28k commits by 199 contributors, >100 pull requests, 621 branches... But the most important thing is that it is very easy to install. I remember when we were building P2PU's Lernanta platform, also on Django, and it was quite difficult to install the code locally (having the right Python version, pulling in all the right libraries, etc), even to make some tiny changes. OpenEdX, which is much more complex, with a number of different services that need to work together, offers several pre-built packages. I chose to use Vagrant to run it locally, and once you've installed Vagrant and VirtualBox, simply using the commands listed here was enough to get a fully functioning local OpenEdX instance:

mkdir fullstack
cd fullstack
curl -L https://raw.githubusercontent.com/edx/configuration/master/vagrant/release/fullstack/Vagrantfile > Vagrantfile
vagrant plugin install vagrant-hostsupdater
vagrant up

Cohorts and single-sign on

Now that I had a functional version of OpenEdX running locally, it was time to look at our challenges. We were interested in two separate but connected pieces of functionality.

enabling single-sign on with external tools, so that students can use an external wiki, for example, without having to create a new account and remember a new password
ways of "scripting" the collaboration process, particularly grouping students into smaller groups, and sending students to various resources based on time, group affiliation, and what they had already done

For the second challenge, EdX already has a "cohort-feature", which let's you divide students into groups, and make certain parts of the discussion forum specific to each group. However, we were interested in much more detailed scripts than that. In our existing course, we used APIs to automatically insert templates into each group's space, take content from different groups and aggregate together, conditionally send students to different tools depending on their previous input, etc. Since EdX does not have an API that gives an instructor live access to the discussion forum, we couldn't for example create 300 cohort-groups and automatically populate each forum thread with a question post, then extract their answers, and do something with them. We would thus need to use external tools for this aspect of the course, bringing us back to the single-sign on issue.

Inspired in particular by the recent EdX MOOC Data, Analytics and Learning, and their inclusion of external social and collaborative tools, I began investigating our options. The simplest way of including external tools, beyond simply linking to them, is to use an IFrame. However, a tool included in an IFrame receives no information from EdX about which student is visiting, and can send no information back.

An intriguing option that the DALMOOC used, is to let the EdX platform function as an OpenID provider, and having a "Login with EdX" button, similar to the many sites that feature "Login with Google/Facebook/Twitter, etc". However, I could not find any information about how to enable this on EdX.org, and it would also mean that the students still would have to do a round-trip back to EdX to allow the login, making it less seamless.

LTI to the rescue

Although I've been working with hybrid and online learning for many years, I've always been drawn to existing Web 2.0 tools, using APIs and RSS feeds to tie things together. Because of this, I don't have much experience with traditional LMSes, and things like SCORM and LTI -- in fact, they always sounded quite cumbersome and complex. However, it turns out that LTI (Learning Tools Interoperability) is a very simple protocol; it's basically a simple web request (either as an IFrame, or as a link opening a separate window) with three extra pieces:

Authentication
The authentication ensures that the request really comes from the OpenEdX platform (in this case), and that I cannot easily fake a request with someone else's user ID.
Student info
The student info in the case of EdX is not any actual student info, but a persistent hashed identifier. This means I can always recognize when the same student accesses the LTI object, and I can also later connect learning analytics from the LTI object with EdX data (there is a hash table connecting the hash IDs with the actual student IDs in the data dumps).
Callback URL
The callback URL is for graded LTI objects, to deliver a simple numeric grade back to the EdX platform.

I first spent some time researching an LTI integration directly with Confluence, but I could not find one that was currently maintained. Limiting ourselves only to web apps that offer LTI integration would also severly restrict our choices, and even if the tools did offer LTI integration, that would only give us something like single-sign on, not any opportunity to implement pedagogical scripts. This led to idea of using an interstitial LTI script. This script would receive the request from OpenEdX, keep track of the relevant information about the student ID, and forward to various web services, providing automatic logon or deep-linking where appropriate/possible.

Early experiment, group-based Etherpad selection

I found an LTI library for Ruby, which had a great little minimal example application. The application is simple -- it prompts the user for a grade, and then sends that grade back to the platform -- but the fact that I could get that to work with OpenEdX using the provided instructions, gave me the confidence to keep experimenting. I decided to use Redis as a backing database, since it is very easy to experiment with, and would also provide easy scaling if needed in the future.

Since we don't know anything about the student arriving, other than the hashed persistent id (there might be ways of getting more information from the system in the future), my first step was a simple form where students would specify their nickname, and choose a group (this could in the future be a long list of groups, or even some complex group formation algorithm).

The next access point then simply forwards to an Etherpad, however it chooses the Etherpad URL based on the group already select (and stored in Redis). This way, we could have three hundred groups, and funnel students into 300 different Etherpads without changing anything in the EdX course setup. We could even have an on-demand feature which puts entrants into a "waiting room" until a pre-determined number of students appear, upon which a new Etherpad room is generated, students are forwarded to that room, and a new waiting room is created.

Confluence integration and templates

Since Etherpad does not use any authentication, it was a much simpler thing to integrate than Confluence. Luckily, Confluence has a rich API, which I've already used extensively in previous courses, however I did not know if it would be possible to automatically log students in. In the end, it turns out that by accessing a particular URL, you can offer as arguments the username, password, and the page to be redirected to:

url = "https://www.WIKIURL.org/dologin.action?os_username=#{wiki_username}" + 
"&os_password=#{wiki_pwd}&os_destination=/display/IN/#{page}".

When a user tries to access a wiki endpoint, I first check in Redis whether the user already has a wiki password. If not, I randomly generate a wiki username and password, and create that user with the appropriate permissions, using the Confluence API. I then redirect the user's browser to the URL above, with the username and password (which they never need to see). The result is that they are seemlessly logged in, and redirected to the page I want.

However, having thousands of users simultaneously edit an unstructured wiki can be chaotic. Our idea is to make it easy for students to supply information, and for a smaller group of engaged users to engage with editing and enriching that information. As an example, I created a simple form which then populates a wiki page. In the current example, upon visiting this LTI object, the student automatically gets a wiki account (if he/she doesn't already have one), the page is created, and the student is automatically logged in and sent to the newly created page. In an actual course, the page could be generated by information submitted by numerous students, and accessing the page wouldn't have to follow immediately after filling out the template.

See it in action

Although this example is quite simple, I find it very promising. I am very happy that due to the OpenEdX platform being so easy to install, and the Ruby LTI library comes with an example app, I was able to get this running in an afternoon. Now comes the hard work -- designing meaningful scripts and interaction patterns for students with very different reasons to participate, and levels of engagement, but at least we know that we can probably implement what we want to. We also need to think about scaling, and whether our local services can handle the amount of users we are expecting.

I'd love to hear about others experimenting with pushing on the interactive and collaborative features in MOOCs and large courses. It would also be interesting to think about how to make LTI tools even easier to integrate -- I'd love to see a catalogue of external LTI tools, ranging from simple individual widgets, to groupware and collaboration tools, which could all be easily slotted into an EdX, Instructure, or even Blackboard course.

All the example code is on GitHub.

Random German Stuff that Matters - great content auf Deutsch

2015-01-23T16:42:31Z

Why introduce German content in English?
Podcasts and radio

Radio programs I like
Podcasts I like

TV, movies and series
Literature

Why introduce German content in English?

One issue that I have not heard much discussion about, is the challenge of finding interesting material in foreign languages. In your own culture and language, you have gotten to know authors through school, through popular media, recommendations from friends, etc. But when you begin exploring a foreign language, it's often like starting from scratch again. I remember walking around in the libraries in Italy, having no idea where to start, or what kinds of authors I might enjoy.

This is made worse by the fact that I am typically much slower at reading in a foreign language, and particularly in "skimming". This means that even though I can really enjoy sitting down with a nice novel, and slowly working through it, actually keeping up with a bunch of blogs, skimming the newspaper every day, or even doing a Google-search and flitting from one page to the next, becomes much more difficult.

Some things are easier these days - it's typically easier to get access to media over the Internet, whether it be movies, podcasts, or ebooks (although this differs a lot between different languages, it is still difficult finding Indonesian movies or TV-shows online, and even Norwegian e-books are hard to come by online), but navigating is still difficult. When I began learning Russian again, I was amazed at the number of very high-quality English blogs that talk about Russian modern literature (places like Lizok's bookshelf, XIX век and Languagehat), which really inspired me to continue working on my Russian, to be able to read the books they so enthusiastically discussed.

These last two years have been a bit of a renaissance for my own usage of German. It's a language I learnt in school in Norway, I read my first novel in it when I was 16, and have visited the country a number of times. However, for the past 10-15 years, I did not have many opportunities to use it, and did not seek out German novels, podcasts or anything like that. However, spending a very enjoyable month visiting a lab at the University of Munich gave me a chance to reawaken my language skills and interest. I then did a short visit last winter, and spent about three months in Berlin this summer.

In addition to the visits, what got me in to reading was ironically Karl May. Not well known in North America, he is the all-time best-selling German author from the 19th Century, writing a large amount of fantastical tales of travel and adventure around the world, from places he had never visited. I had heard about him, but never read anything, and decided that I would try it out for fun. I found "Deutsche Herzen, deutsche Helden" as a free download and began. It was certainly a fantastic tale, taking us from Istanbul, to Egypt, through the Wild West and ending up in Siberia.

Long like a Bollywood-movie, it felt like each part could have been it's own substantial book. Full of handsome and brave Germans, it was certainly not always politically correct (although I have read far worse things in English from the colonial times), but quite enjoyable, and at the end, I realized I had read almost 3000 pages (these things happen on a Kindle), and that my German reading speed had improved measurably as a result.

Since then, I have found a great number of enjoyable German novels and podcasts, and I wanted to share some of these with you. If you speak German as a foreign language, but were not quite sure where to start, perhaps some of this will be useful. And if you don't speak German, perhaps you'll be inspired to learn. Of course, this is just a sampling of what I've randomly come across, not an exhaustive survey, and it unapologetically is stuff that I like and enjoy, no attempt at objective critique.

Podcasts and radio

I love going for walks and listening to podcasts and audio books. I've also found that audio supports language learning and maintenance in a different way than reading; in the same way that reading a lot can make you a better writer, I find myself speaking more fluently and confidently after listening to audio in a language, even if I haven't actually spoken it for a long time. A good example was when I met someone speaking German in a Moscow airport a few years ago. At that time, I hadn't been speaking German for years, but just finished listening to Momo, and was surprised by my fluency.

Germany has a very vibrant private podcasting scene, and also very high-quality TV and radio stations. Most of these have so-called Mediathek, where you can easily find programs and links to feeds (like ARD's Mediathek).

Radio programs I like

DRadio Wissen Einhundert

Einhundert is my favorite German radio program. Every week, they choose a theme, which can be very broad ("Starting over", "Hitting the road"), but also more specific (a program about when half a million Russian soldiers left Eastern Germany). In each case, they find people with interesting stories that somehow relate to the theme (often in very different ways).

It's a bit similar to This American Life or RadioLab (the latter of which I love as well), but they have managed to find their own voice. It is difficult to put the finger on what is so brilliant about this program, but somehow the combination of interesting stories and good editing makes for great listening. I often tell my wife about stories I head in this podcast, two of the more memorable ones:

News

You can watch or download the video of the main evening news in Germany, Die Tagesschau, and sometimes I watched the news on my iPad while doing dishes. However, my preference is to download the audio and listen while walking. Surprisingly, I find I never really miss the visuals. I appreciate the focus on domestic and international politics, rather than "a man was hit by a bus yesterday".

Listening regularly, you get a good understanding of the main political issues in Germany and the EU, and how they are developing. It's been particularly interesting following the Ukraine-conflict through German eyes. I also often listen to Tagesthemen, which tends to go a bit more in depth. There is also Bericht aus Berlin, which is a weekly political magazine.

Other academic programs

DRadio Hörsaal is an interesting format, where they first interview a researcher about their research, followed by a public lecture made by the same person. The topics vary widely, and I remember hearing a fascinating talk about how we now play classical music faster, than when it was authored.

SWR Wissen is a series of traditional radio documentaries, where a journalist tells a story, intermixed with some interviews. Topics vary very widely, but somehow the good story-tellers are able to make many topics that don't sound very enticing into interesting experiences. (Looking at the site now, I see that they also make transcripts of all programs available in epub-format - quite innovative.)

Podcasts I like

Chaosradio

Chaos Computer Club is a historically famous network of hacker-clubs in Germany, and their Berlin chapter produce a monthly radio program/podcast called Chaosradio. The goal is to explain technical concepts to a more lay audience, so they usually have members of the club moderated by a radio moderator. The shows last something like three hours, and it can be quite enjoyable to hear the friendly bantering of the participants. For example, I quite enjoyed the show last winter about the NSA surveillance scandal and how mass-digital surveillance works.

CRE: Technik, Kultur, Gesellschaft

Through Chaosradio, I came across CRE, which began as an extension to Chaosradio, but now has a life of its own. It's run by Tim Pritlove, who is an incredibly prolific German podcaster, and the format is basically Tim sitting down with someone who is a specialist on a certain topic, and just talking. Sometimes the conversations will go over 3-4 hours, and somehow listening to two very intelligent people in a nice conversation can be quite interesting and calming. Some of my favorite episodes:

Wikidata, the new semantic Wikimedia project, and how we think about ontologies and hierarchies to organize knowledge
Stadtplanung, city planning from Alexandria and Babylon to Russian suburbs, Syria and Berlin
Bundeswehr, the role of the German army in international conflicts

Tim also hosts a few other noteworthy podcasts, including Logbuch:Netzpolitik, which reminds me of many English-language podcasts, where two people sit around every week and chat about "what's new" related to Internet and technology politics. A more structured interview-podcast is Fokus Europa which attempts to explain the structure of modern Europe, focused mostly on the European Union. I found the first two episodes Geschichte der Europäischen Einigung and Deutschland und Europa quite interesting.

Staatsbürgerkunde is a great example of oral history. Martin Lutz interviews his parents Martin and Christine Lutz about their life in the DDR, mixing general knowledge and history with their own personal recollections and anecdotes. I particularly enjoyed the episode on Weissensee, the drama series I mention below.

TV, movies and series

This will unfortunately be a very short section, and I would love further suggestions. I enjoy watching German movies when I come across them, but I have no good online source to find them (Netflix only has a few). I have also been looking for good German drama series, and would love to find something like the great Danish crime dramas (like The Bridge). However, I'll mention one great drama series and a serie of documentaries.

Deutschland von oben is a series of documentaries using a combination of satellite and aerial footage to show different aspects of the geography of Germany. There are several themes, "City", "Land" and "Water", and we go from looking at the majestic golden eagles in Berchtesgaden, to how current street patterns are the result of the structure of the initial Roman settlements. It gave me a much better understanding of the diversity of German landscapes and regions, and although it can get tiring, the dramatic music and voices go well with the sweeping vistas. All the episodes are available in streaming HD, and there is also a feature-length cinema version.

Weissensee is a mini-series in two seasons about life in East-Berlin during the DDR-times. The series focus on two families, a Stasi-family and a rebellious artist family, whose lives interact in various ways. There are definitively aspects of Romeo and Juliette, and cliches from other mini-series, but it's very well done, and apparently very period-authentic. I very much enjoyed both seasons, and also the discussion about the series on the Staatsbürgerkunde podcast mentioned above.

Literature

Reading fiction is one of my main ways of approaching a foreign language, and German is a very welcoming host. It has become much easier now that we have access to e-books, and literature portals. I remember when I began learning German and Italian 15 years ago, I would go into a library and be completely lost as to where to start. These days, we have great sites like Krimi-couch, or even the literature section in Die Zeit for more high-brow literature.

In general I quite enjoy crime novels, and I found a German crime writer tradition that felt quite similar to the Scandinavian crime that I grew up with. It tends to be tightly bound to a specific region or town, and to spend just as much time on the personal lives of the investigators, as well as the perpetrators and victims of the crime, sometimes also touching upon societal trends and challenges. This is not the hard-boiled crime noir from smokey San Francisco PI-offices, but rather a great way to examine different tensions in society, different occupational groups, life in small and big cities, etc.

One of the things I have really missed in Canada, is regionally based crime stories. I think Toronto has many stories waiting to be told, and not just Toronto, but specific areas - Scarborough, the Junction, white-collar crime on Bay street, corruption at Queens Park, etc. In Germany, the concept "regiokrimi" (regional crime) even has it's own website. Apparently there are even small towns that commission crime novels set in their area for the town's anniversary. This has led to some critical voices, complaining that literary aspects are overlooked in an attempt to stuff as many local references as possible, but I've come across a number of authors that I really enjoyed.

Klaus-Peter Wolf has a fascinating history on Wikipedia. Apparently he almost works like an investigative journalist, often diving deep into the various thematics that he then fictionalizes, going so far as to set up a fake mailorder-marriage company before he wrote a book about the trade with women, and living with a youth gang, before writing a book about youth criminality and violence. However, my favorite is his series of novels from Ost-Friesland, an area I had never heard about before, and now feel that I know well enough that I could navigate the area without a map.

Another set of great novels set in the area come from Sven Koch.

Jaques Berndorf writes about another area of Germany that I was not familiar with, the Eifel. In this case, the protagonist is an investigative journalist, rather than an investigator, and somehow he always gets tangled into complex issues, often involving the state and the secret services. A cranky old man living alone in the countryside, there always seems to be an intriguing female showing up just in time to join him in his investigations.

Reminds me of some earlier (80's-90's) Swedish crime, which also talked about investigative journalists, the Swedish secret-services and government, and their connections to foreign powers, etc.

Jenk Saborowski writes about the fictional European Federal Police, and agent Solveigh Lang, who travels Europe to catch criminals. Intriguing fiction, which reflects a modern Europe connected with high-speed rail and the absence of borders, but still divided by different bureaucracies, languages and cultures.

Tom Liehr has been compared to Nick Hornby, and it's a comparison that makes a lot of sense. He writes about young men, fixated on music, DJing, being a radio host, or going on crappy chartertours. Great writing, and very rich characters.

There are so many more authors I could list, here are some very quick introductions

Nele Neuhaus: Decent crime set in "Vordertaunus", rural area near Frankfurt. Somehow always ends up with a situation in which a large number of people could have had motive and possibility to kill the victim.
Sebastian Fitzek: "Psychodrama"-bestseller. Really enjoyed Passagier 23 and Amokspiel, but others of his novels are too "psychological", with the I-voice seemingly going crazy, unable to believe anything he sees/remembers, etc.
Petra Durst-Benning: Interesting and enjoyable historical fiction from various periods in Germany, usually centered around strong women.
Birgit Schlieper: Very interesting and well written novels about difficult issues facing young people, with a strong psychological aspect. Powerful and thoughtful.
Katharina Peters: Fascinating crime novels from the other German coast, Rügen, often involving the Nazi-history of the area.
Martin Suter: A series of crime novels, and some free-standing novels, usually dealing with the Swiss upperclass. Charmingly written. Best free-standing novel is Der Koch, amazing writing (although it looses some of the steam after the beginning).
Sigrid Ramge: OK crime stories set in Baden-Württemberg, but felt like very "regio-krimi", almost like she wants to give you a detailed tour of the area, rather than just building it into the story.
Manuela Kuck: Well-written crime from the Wolfsburg-region.
Sebastian Lehmann with Genau mein Beutelschema, hilarious and well-written ironic take on the different areas and subcultures in Berlin. Perfect reading while staying in Neukölln last summer.

Gute Vergnügen!

Starting data analysis/wrangling with R: Things I wish I'd been told

2014-10-14T16:17:31Z

Use RStudio
Use Knitr
Learn from others
Separate cleaning and organizing from analysis
Use version control
Learn the DSLs in R
DRY - Don't repeat yourself
Keeping track of types in R
Asking questions - providing reproducible examples
Packages to use

R is a very powerful open source environment for data analysis, statistics and graphing, with thousands of packages available. After my previous blog post about likert-scales and metadata in R, a few of my colleagues mentioned that they were learning R through a Coursera course on data analysis. I have been working quite intensively with R for the last half year, and thought I'd try to document and share a few tricks, and things I wish I'd have known when I started out.

I don't pretend to be a statistics whiz – I don't have a strong background in math, and much of my training in statistics was of the social science "click here, then here in SPSS" kind, using flowcharts to divine which tests to run, given the kinds of variables you wanted to compare. I'm eager to learn more, but the fact is that running complex statistical functions in R is typically quite easy. The difficult part is acquiring data, cleaning it up, combining different data sources, and preparing it for analysis (they say 90% of a data scientist's job is data wrangling). Of course, knowing which tests to run, and how to analyze the results is also a challenge, but that is general statistical knowledge that applies to all statistics packages.

So here are some of my suggestions and "lessons learnt", in no particular order. Some will find the code samples scary, others will find the suggestion to use for-loops far too basic, but hopefully you will find something useful here.

Use RStudio

RStudio is an great open source integrated development environment for R. It is free and open-source, and integrates

a project manager
a text editor with syntax highlighting and tab-completion
package management
previewing plots
previewing tables (see columns and rows in your dataset)
integrated help (click F1 on any function)
jump to source (click F2 on any function)
and Knitr (write reports in Markdown, see below)

There's even a version of RStudio that runs in the browser – we're currently using it to coordinate data analysis among a geographically dispersed team on many gigabytes of data. Keeping it on a central server, and letting people run analyses directly on the data is much more convenient and secure than having everyone store tens of gigabytes of data on their personal computers.

Use Knitr

Literate programming is the idea of mixing executable code with documentation in the same document. Knitr brings this functionality to R, and it's integrated beautifully with RStudio. By default, all the text you write in a Knitr document is interpreted as Markdown (a light-weight markup language, which I'm also using to author this blog). Press Alt+Cmd I to insert an R code block. You can run the code either by pressing Cmd+Enter on a single line, or Alt+Cmd C to execute an entire code block. When you're done, you can press "Knit HTML", which executes the whole document and produces a report.

Here's an example from some recent work analyzing a questionnaire, we're introducing the graph, and then adding the code that will produce the graph (see my previous blog post on likert-graphs):

Running Knit HTML combines the text, formatted nicely, the code used to generate the graph, and the actual graph:

For the final report, you might choose to hide all the code segments with this invocation:

library(knitr)
opts_chunk$set(echo=F, warning=F,message=F,results="asis", prompt=F, error=F)

Not only is this a great way of writing reports (you can also export to PDF, or even write in LaTeX instead of Markdown), but it's a very nice way of organizing your code. I now do all my development in this mode, even for scripts where I'm not interested in the final report. I like the ease of documenting, the clear visual separation of the different code blocks, and the ease of pressing Alt+Cmd C to execute a single code block.

I recently gave a small demo to a research group, showing them RStudio, using Markdown to write knitr reports, and Shiny to make interactive webpages:

Learn from others

There are lot's of R textbooks and documentation out there. Two other great sources of ideas are RPubs and r-bloggers. Using knitr to write reports in RStudio, as described above, you have the option to upload your report to RPubs. You can also see all the reports that others have written. These are great to learn from, since they typically contain both all the code, and the output and graphs, tying it all together into an analytical narrative. Unfortunately, there is no good way of searching the site, so you will find people's homework nestled between great expository writing, but it's still well worth a visit. You can also visit the knitr notebook showcase to see some select examples.

R-bloggers is an aggregator for blog-posts about R and statistics, and it's a great way to discover new packages or R features, and see how other people attack various data analysis challenges, often using publicly available datasets.

Separate cleaning and organizing from analysis

Although R is a fully-fledged (although a bit crufty) programming language with object-orientation and functions, the code that users typically write is very different from an R package. Users usually write very imperative code, "load this file, then transform the second column, then add the third column, then graph it". However, acquiring some good habits of organizing the code and working with the data, might save you a lot of time in the long run. (Also see the section on DRY below)

I usually separate data importing and cleaning from the analysis. My goal is to leave the raw data completely unchanged, and do all the transformation in code, which can be rerun at any time. While I'm writing the scripts, I'm often jumping around, selectively executing individual lines or code blocks, running commands to inspect the data in the REPL (read-evaluate-print-loop, where each command is executed as soon as you type enter, in the picture above it's the pane to the right), etc. But I try to make sure that when I finish up, the script is runnable by itself.

Knitr helps impose this - when you choose Knit HTML, it begins with a clean slate. When you are working in RStudio, you might have objects lying around from calculations you did a while ago (with code that you've already changed), but if the "knitting" is successful, you know that the current code is valid and produces exactly what you see.

Example of preparing data

In one example, I had gotten spreadsheets from several students who had helped me enter data from a large survey. I could have opened these in Excel, and copied and pasted the information into one sheet, but instead I left the files as they were, and read them into R. I tag the columns from each spreadsheet with provenance, if I want to run any quick tests, or even more formal interrater reliability tests, and merge them into one table. In some spreadsheets, there was an extra empty column, so I remove that programmatically (rather than editing the Excel spreadsheet). First I load xlsx to read the Excel spreadsheets, and plyr to rename fields later.

library(xlsx)
library(plyr)

Then I read in and join the spreadsheet files:

ana <- read.xlsx(file="ana.xlsx", 1,stringsAsFactors=FALSE)
ana$by <- "Ana"

chad <- read.xlsx(file="chad.xlsx", 1,stringsAsFactors=FALSE)
chad[["NA."]] <- NULL
chad$by <- "Chad"

dd <- read.xlsx(file="DD.xls", 1,stringsAsFactors=FALSE)
dd$by <- "DD"

db <- rbind(ana, chad, dd)

I noticed that some of the spreadsheets had a bunch of empty rows at the bottom. This might not be the most elegant way, but I used this function to remove all the spreadsheets with all empty values. The way this works is that is.na(x) produces a list of "TRUE TRUE FALSE FALSE TRUE" depending on which of the columns has an NA for that particular row, and sum() adds up the TRUE's as 1, and FALSE's as 0.

db2 <- db[apply(db,1, function(x) {
  sum(is.na(x)) < 43}),]

Then I move on to turn all the various versions of "empty cell" into NA:

db <- as.data.frame(lapply(db2, function(x){
  x <- replace(x, x %in% c("n", "N", ""), NA)
  x <- as.factor(x)}))

Some of the columns are demographic values, so I'll change these from numeric to categorical variables with the actual values:

db$gender <- revalue(db$X1, c("1"="Male", "2"="Female", "3"="Trans", "4"="Other", "5"=NA))
db$major <- revalue(db$X2, c("1"="History", "2"="Religion", "3"="Other"))

And the rest of the questions are likert-style questions with the same categories, so I'll both rename them and order them in one swoop:

likertcat <- c("1"="Not at all", "2"="To a small extent", "3"="To some extent",
  "4"="To a moderate extent", "5"="To a large extent")

for(e in names(db[,9:44])) {
  db[[e]] <- revalue(db[[e]], likertcat)
  db[[e]] <- ordered(db[[e]], levels= c("Not at all","To a small extent",
    "To some extent","To a moderate extent","To a large extent"))
}

I also add categories and groupings using my own addition, which you can read more about, and finally I save the new table as an RData file:

save(db, file="db.RData")

I should always be able to re-run this script, and re-generate the exactly same RData file, but typically the only time I'll re-run the script is if I get additional data (let's say two other students send me their data entry files).

I then begin the data analysis script with

load(file="db.RData")

And we have a nicely prepared table that we can begin exploring.

Use version control

RStudio comes with support for git baked in, and it's a great practice to use it. When you create a new project, check the box for "Create a git repository for this project", and you get all this functionality for free. Once you have a functioning version of your script, commit it to Git, and when you later make changes, you can easily track those changes back in history, restore earlier versions, etc. This is great for people collaborating, but it's actually a great idea to start doing it even on solo-projects, and can save you a lot of grief in the future. RStudio has a good writeup of this functionality.

Of course, if you are able to do so, it would be great if you could share your scripts (and data) on GitHub or another public repository. Even in cases where you can't share your raw data (because of ethics, etc), your code could be useful to others. (I have shared some of the code we used to analyze Coursera MOOC data in a GitHub repository, but I could probably go through my folders and find more snippets of code to share).

Learn the DSLs in R

DSL stands for domain-specific language, and the key to being efficient in R (and one of the reasons beginners might feel that the learning curve is very steep), is to realize that R is actually composed of several different embedded languages. Most end-users of R actually don't need to know very much about the R programming language, mostly people write scripts in a very imperative way "first do this, then do this, then do that". There are probably data analysts that have used R for years, and never written a function or a loop statement.

However, if you want to plot graphics with ggplot, you will have to learn an entirely new way of thinking. It is based on the Grammar of Graphics, and makes it possible to intuitively build up exactly the kind of graph you want, based on layers, mapping data to geometries such as bar-charts, etc. However, if you don't understand the underlying logic, even making the simplest plot will seem like a strange incantation.

Similarly, for data wrangling, there are several options. The plyr family of functions is very powerful, for example here is how you would take a data.frame with per-country data, and calculate per-continent summary statistics:

ddply(gDat, ~ continent, summarize,
      minLifeExp = min(lifeExp), maxLifeExp = max(lifeExp),
      medGdpPercap = median(gdpPercap))

An alternative is to use data.table. Data.table is an alternative to the built-in data.frame, which is much more performant for large datasets, and also offers powerful indexing, transformation/grouping, and merging/joining. It has a very dense syntax, which is powerful once you understand the logic. Here is an example of calculating a number of statistics on a football dataset:

dt.statsPlayer <- dt.boxscore.subset[,list(minutes=sum(minutes),
                                           mean=mean(pp48),
                                           min=min(pp48),
                                           lower=quantile(pp48, .25, na.rm=TRUE),
                                           middle=quantile(pp48, .50, na.rm=TRUE),
                                           upper=quantile(pp48, .75, na.rm=TRUE),
                                           max=max(pp48)),
                                     by='player']

(Note also the use of white-space, R often let's you space expression out over several lines, which can help a lot with readability).

The key to getting proficient in R is to choose one main way of doing things, for example ggplot2 for all graphics (and not worrying about learning grid and lattice plotting), and either data.table or plyr, and focusing on understanding their underlying logic, and using them frequently.

DRY - Don't repeat yourself

This used to be a mantra when programming Ruby, but is often overlooked in R code. Since many people think of doing analysis in R as simply writing a number of instructions to be executed linearly, we forget that the language let's us be much more efficient. I did say above that you don't need to understand the R programming language in depth to use R profitably, however, a few features are very useful.

The advantage of avoiding repetition is that your code becomes easier to read (because your intention stands out), and it becomes much easier to reuse code, and to update. Let's say you are plotting five or six similar plots, and in each case you have a big blob of ggplot2 code, with layers, themes, etc. Now your boss tells you to change the theme of all the plots. Rather than jumping around and editing a bunch of lines, perhaps introducing new errors, you could try to abstract out the plotting code to a function, which you'd only need to edit once.

Turning often used code into functions

Here's an example of a simple plot I used frequently in an exploratory report I wrote about MOOC data. I first clean out the NAs from the dataframe, and then display it as a colored bar chart, where each bar corresponds to a region, and the colors correspond to age groups. The first line defines the plot, and the two subsequent lines formats it (flipping the axis, removing some clutter, adding a legend title). (Click the image to the right to see the full graph.)

dbgen <- db[!is.na(db$gender), ]
ggplot(data=dbgen, aes(region, fill=age)) + geom_bar(position="fill") +
  coord_flip() + scale_y_continuous(labels = percent) + ylab("") + xlab("") +
  guides(fill=guide_legend(title="Age"))

Since I needed a few graphs, the initial impulse would be to copy it and modify a few parameters. But how can we instead make this into a generic function? Turns out that in general we'd want to remove NAs from both the grouping and the fill column (the region column just didn't happen to have any NAs, so we didn't need to check that above), and we also need to switch to using aes_string because of how ggplot implements its "DSL", otherwise it's quite simple.

We define a function my_sideplot, which takes the full dataframe, the grouping variable, the fill variable, and an optional title for the fill legend, and generates a plot similar to the one above.

my_sideplot = function(db, group, fill, fill_title="") {
  db = db[!is.na(db[[fill]]) & !is.na(db[[group]]),]
  ggplot(data=db, aes_string(group, fill=fill)) +
    geom_bar(position="fill") + coord_flip() +
    scale_y_continuous(labels = percent) + ylab("") + xlab("") +
    guides(fill=guide_legend(title=fill_title))
}

The advantage is that I can now very easily experiment with different versions. Perhaps using the age variable for grouping, and the regions for fill is better?

my_sideplot(db, "age", "region", "Regions")

This makes my code much easier to read, since I see right away that this is doing exactly the same as the code above (otherwise you need to look very closely at the ggplot code to see what is different), and if I have a knitr report with 10-15 of these graphs, and I want them all to use theme_bw() – a black-and-white theme – I can simply modify the function, and rerun the graph.

Functions can of course be much more advanced than this, with branches and sanity checks, etc. However, as a first approximation, just taking chunks of code that are repeated with very few changes, identifying the changes and making them into parameters, and turning the code chunk into a function, can already be very useful.

Using lists and iteration

The second approach to cleaning up code is to use lists and for-loops. For example, we might want to generate plots of six different categorical variables grouped by region. Or perhaps we want them grouped by region, age, and English-skill? Now we are talking about a total of 20 graphs. We could copy and paste the one-line my_sideplot invocation above, which is already a big improvement, but we can do much better.

group_vars = c("region", "age", "english.skill")
fill_vars = c("usercat", "education.level", "degree.program")
for (groupv in group_vars) {
  for (fillv in fill_vars) {
    my_sideplot(db, groupv, fillv)
  }
}

This code will iterate through the group_vars, and for each group_var, it will iterate through the fill_vars. For each fill_var, it will render an appropriate plot.

You can also iterate through for example the columns in a data.frame. Let's say we want to turn all the character columns into factors:

for (e in names(db)) {
  if (is.character(db[[e]])) { db[[e]] <- as.factor(db[[e]])}
}

Keeping track of types in R

Again breaking against my statement that you don't need to know much of the R language, but the various data types is one of the things that has tripped me up the most. This is because certain operations can return different data types than you were expecting.

There are many R tutorials, and I am not going to give a full introduction here. The most basic datatype is the vector, and one of the peculiarities of R is that even single numbers or strings, are vectors of size one.

identical(c("hi"), "hi") # => TRUE
identical(c(1), 1) # => TRUE

A dataframe is a collection of vectors. Sometimes when you do an operation on a data.frame, you might get a vector back, or a matrix. You can usually cast one data type to another with as.*, for example as.data.frame()

> as.data.frame(list("A" = c(1,2,3), "B" = c(2,3,4)))
  A B
1 1 2
2 2 3
3 3 4

You can check the class of an object with class(),

> class(db)
[1] "data.frame"
> class(db$age)
[1] "ordered" "factor"

If you run into weird bugs, make sure that every step of the pipeline returns the data type that the next function is expecting (or explicitly cast). Also remember that certain functions work differently depending on different data types. For example, length() on a list or vector returns the number of elements. length() on a data.frame returns the number of columns, if you want the number of rows, you need nrow().

Asking questions - providing reproducible examples

There are many great resources for getting help with R, including all the books I showed pictures of at the beginning of this post. An advantage of R being text-oriented is that it is easy to paste exactly the code that is misbehaving, and for others to help out. StackOverflow has a very active R community, however you have a much higher chance of getting help if you make your problem reproducible.

Sometimes you can use the built-in data sets in R, which you can list with the command data(). For example, with the command data(swiss), I load a dataset with data on fertility and education in Swiss cantons. I can use this to illustrate a problem, and anyone else using R will have the same dataset, and can perfectly reproduce my problem, or check that their solution works correctly.

If none of the built-in datasets work, you can create a minimal example dataset that illustrates your problem. Let's say we had a problem with the a data.frame we constructed above. The command dput(a) gives us

structure(list(A = c(1, 2, 3), B = c(2, 3, 4)), .Names = c("A",
"B"), row.names = c(NA, -3L), class = "data.frame")

We can now construct our example:

a = structure(list(A = c(1, 2, 3), B = c(2, 3, 4)), .Names = c("A",
"B"), row.names = c(NA, -3L), class = "data.frame")

ggplot(a, aes(x=A, y=B)) + geom_point()

...and anyone can paste this into their R interpreter, and see the same output that you get. Longer explanation and here's a small example.

Packages to use

One of the strengths of R is the incredible selection of packages available, but this can also be quite disorienting to a beginner. Here are just a few packages that are often useful:

Package	Description
plyr	Plyr applies the "split-apply-combine" approach to R data. You might want to calculate the average height of players, grouped by nationality and year of birth - a one-liner in Plyr. Hadley Wickham is currently developing dplyr, a faster and more powerful version
Shiny	Shiny let's you make interactive graphical websites with R (similar to the way we created the function above, by substituting the parts of the code that vary with variables, and letting the end-user choose the variables). Also see the video above
data.table	As mentioned above, a much faster version of data.frame, very well suite to larger datasets, with powerful split+apply, merge, etc. functionality offering an alternative to plyr
likert	If you ever deal with likert-style questionnaire data, this package makes it very easily to visualize the results. See also my addition to this package

Hope that was useful. Once you want more, you could do worse than checking out Hadley Wickham's Advanced R book. There's also an incredible amount of good textbooks, examples, blog posts, etc.

A pedagogical script for idea convergence through tagging Etherpad content

2014-10-03T21:17:26Z

I describe a script aimed at supporting student idea convergence through tagging Etherpad content, and discuss how it went when I implemented it in a class

Background

In an earlier blog post I introduced the idea of pedagogical scripting, as well as implementing scripts in computer code. I discussed my desire to make ideas more "moveable", and support deeper work on ideas, and talked about the idea of using tags to support this. Finally, I introduced the tool tag-extract, which I developed to work on a literature review.

Context

I am currently teaching a course on Knowledge and Communication for Development (earlier open source syllabi) at the University of Toronto at Scarborough. The course applies theoretical constructs from development studies to understanding the role of technology, the internet, and knowledge in international development processes, and as implemented in specific development projects.

I began the course with two fairly tech-centric classes, because I believe that having an intuition about how the Internet works, is important for subsequent discussions. I've also realized in previous years that even the "digital generation" often has very little understanding of what happens when you for example send a Facebook message from one computer to another.

So I spent the first week focused on the physical infrastructure of the Internet, the history and development, protocols and structure. We used traceroute to explore connections between local computers, from Scarborough to downtown Toronto, or to servers in the US or China. We used telnet to simulate a connection to a web server, we handwrote a very simple webpage, and I had students in groups "draw" the process of sending an e-mail from Toronto to China (inspired by John Britton's idea in his P2PU course)

The second week, in which I implemented this script, was focused more on the software side, particularly Web 2.0. I wanted the students to understand the history of the Web, the concept of the social web, how websites have gone from simple containers of content to web apps and places of interaction, the interaction between commercial providers and user-generated content, etc. I also thought this would be a great opportunity to try out some interactive production and organization of knowledge in practice.

The script

My basic idea was to have the students work in small groups on adding ideas to group Etherpads. My key question was: What are the distinguishing features of Web 2.0. I would have a few different "external stimuli", after which they would add more ideas to their own pad. At the end of the class, we would use tagging to create a folksonomy, and then collaboratively reduce the folksonomy to a few key categories. Based on the categories, my script would extract the tagged pieces of text from everyone's pad, and create one new pad per category. My idea was then to assign the students to edit these pads and turn them into wiki articles that students could later refer to when studying, or working on the final project.

Inspired by Pierre Dillenbourg's work on orchestration graphs, which he talked about at ICLS in Sydney and LASI in Cambridge, MA, I attempted to visually represent the script.

Script (click for a bigger version)

The graph is organized across three vertical levels, representing work done by students individually, in small teams (in our case groups of 2-3 students), and done in the whole class. The x-axis represents time, and the stars represents processing by an external script which shuffles information between pads. Below, I'll discuss the script more in detail.

Phase 1

The students had been assigned two readings as homework, and came to the class prepared. In addition, they mostly had pre-existing ideas about Web 2.0, the social web, etc. Before doing any lecturing at all, I asked them to gather into groups of 2-3 people (based on where they were sitting), and counted off. A script had already generated a number of Etherpads with identical prompts, that were all linked from the week's wiki page.

Phase 2

The idea was to have a few rounds of providing the students with stimulus that would lead to the generation of more ideas, which could be recorded in the pads. After the students had finished their initial brainstorming, I led the class in a discussion, talked through the articles, and showed two Michael Wesch videos: The Machine is Us/ing Us and Information R/evolution. I then gave the students some time to write down ideas and thoughts.

I wanted to give the students a sense of how the web has developed, and I've found the Internet Archive Wayback Machine a great tool in the past. This let's the students explore the history of websites they are familiar with. We looked at a few examples together, and then I asked each group to choose a few websites they were familiar with (and encouraged them to also look at websites in other languages, since we have a very multilingual student body), and specifically look at features and functionality that has changed, not just design. We then discussed some of these examples together, and the students added more ideas to their pads.

The final stimulus was looking in-depth at a specific Web 2.0 site. I had prepared a list of URLs, and the script automatically inserted a random URL at the bottom of each group's pad. The URLs included Wikipedia, Flickr, a newspaper, the university LMS, etc. I always enjoy walking around the room when students are working on these kinds of tasks, because the quick conversations we have while they are in the middle of exploring a topic, and I am able to point them in the right direction, seem really valuable. (And something I have not really captured in the orchestration graph).

Phase 3

Individual tagging

At this point, each student group had a pad filled with notes and ideas about Web 2.0 features and functionality, the shift from Web 1.0 to Web 2.0, etc. They had gradually added to this pad during the class, and I had provided rich external input, and opportunities to discuss in small groups, and in the whole class. If the class had ended here, I would have already considered it a success.

However, one of my design goals for this iteration was to make content generated in class more accessible and useful to students going forwards. Nobody would probably wade through a bunch of unorganized Etherpads when they were later working on the final project, or preparing for the exam. I wanted to see if my tag-extract approach, which had been so useful to me in organizing and structuring my own notes and ideas when doing a literature review, could help the students collaboratively reorganize all of their notes into a coherent whole.

I first asked the students to add tags to the ideas in their pads. They were all familiar with tagging from Twitter, and from the readings and videos in class, and I connected the exercise to this, without telling them about the next step yet.

Below you can see an example of the evolution of the pad from one student group, ending with the students adding tags to their existing ideas.

Collectively organizing the folksonomy

After the students had added tags to their pads, the script extracted all the tags, and created a new Etherpad listing them all. Including misspellings and variants, the students had come up with almost 100 tags. The pad with all the tags offered a simple way of categorizing tags, if you have a list of tags that are similar and should be grouped together, such as this list from our class:

advertisments
viralmarketing
consumerism
marketization
onlinerevenue

We just needed to combine them together into one category:

advertisments: viralmarketing, consumerism, marketization, onlinerevenue

Because we were using Etherpad for this as well, the whole class could participate, and in just a few minutes, we had organized the tags into a much smaller number of categories. In the example above, this was quite easy, but for others, the ordering might have been a bit arbitrary. Since we did not have access to the actual text tagged at this point, it was also sometimes hard to know what people had meant.

See the animation below of this process.

Putting it all together

The final move was to run a script which would go through all the pads, and extract all the text tagged into appropriate category pads (ie. above, text tagged either #advertisments or #viralmarketing or #consumerism would all end up in a pad called advertisments). There was a bit of an anti-climax as that script failed to run at the end of the class, because of a silly error. That was OK, because when I fixed the script after class, and ran it, I realized that the output was often not as useful as I had naively hoped.

On the right, you see an example of a pad, with the title listing all the tags in the category, and the specific category also appended after each text snippet. (I could also have appended the name of the group pad that the snippet came from, but in this case, that would just make it more cluttered.)

Evaluation

The first part of the script worked great, but was not that different from the way I often teach. Pushing different URLs to each pad worked very well (and students loved having something suddenly show up beneath the text they were just editing), I've used this particular script again in a later class.

However, the grand idea of using tags to reorganize and structure the text still needs more work. I think the prompt was the first problem, "What distinguishes Web 2.0" was not perhaps the most productive or clear question I could have asked. I could have made it more problem-oriented, or debate (pro/con), etc. I also never explained to the students the purpose of what we were doing - they are used to taking notes in the Etherpad during group work, but I never told them that it would be tagged, cut up into small pieces, and shared.

Now do this, then do that

Once we tagged, I didn't tell them that we would gather all the tags and organize them, and while we were organizing the tags, I didn't tell them what we were doing it for. I think teachers often do this unconsciously – lead students through activities that they have clearly thought out, without telling the students where they are going, or why they are moving in a certain way. Sometimes this is a necessary part of the script, whereas at other times, students need conscious training in certain epistemic moves, to be able to carry them out effectively.

I was very impressed by Kate Bielaczyc's presentation at ICLS 2010, where she used a sport metaphore to distinguish between practice and playing a match. To get good at a sport, you don't just play matches, the coach leads you into targetted practice, setting up contrived situations to challenge you. Similarly, she had her students practice the act of knowledge building with contrived situations, before they actually got into authentic knowledge building.

Hypothetical game-configurations are used to reflect on the knowledge building moves made possible by a particular configuration of knowledge objects. The configurations consist of “snapshots” of hypothetical student work in Knowledge Forum, meant to capture game play at a fixed point in time in order to engage the community in asking: given this configuration, what types of knowledge building moves would best contribute to advancing our knowledge? Some configurations focus on single moves, such as presenting a possible initial explanation generated in response to the problem the students are working on.

Students then generate a knowledge building move meant to advance this initial idea. There are also more complex game configurations that present not only a possible initial explanation in response to a problem, but also provide a series of possible knowledge building moves. In this case, students both evaluate the quality of the provided moves and generate a possible next move that contributes to the progressive improvement of ideas. In all cases, students each work on the same hypothetical game-configurations so that they can then compare and contrast their proposed knowledge building moves in whole-class discussions about issues such as what makes a “good contribution” and what does it mean to advance the community understanding. (Bielaczyc & Ow, 2010)

So perhaps if I did this again, with a better prompt, and students knew that they were taking notes that would be cut apart and reassembled, they would write differently. However, it is also possible that this kind of reorganization is not as useful for collaborative work – part of why it works when working on my own ideas, is that I wrote the notes myself, and I remember why I wrote them. Looking at other people's notes is difficult at best, and especially when they are out of context.

Future work

I still believe that Etherpad and the wiki, which is also scriptable, has a lot of potential for rapid experimentation with collaborative scripts. We have experimented with a number of other scripts in the two classes that I teach/co-teach, and I might blog about other ones later. I've used a simplified version of the tagging script in two later classes, where I focused on the ability to pick out and aggregate ideas based on pre-determined tags. The fact that Etherpad records the entire history, and let's me go back, is also useful for research, and I plan to spend a bit longer looking at how the individual team pads evolved during the different stages of the script.

The week after the script above, we talked about theoretical perspectives on development and technology, and as an ice-breaker before the lecture, I asked them to write down what they thought poverty was, and what they perceived the "official" definition to be. After some group discussion, I asked each group to come up with a single sentence for each, and tag them with respectively #ithink and #worldthinks. The script then simply aggregated all of these on a simple webpage, which was a great way of exploring different approaches (ethics, capability approach, materialistic, relative/objective poverty, happiness, etc).

Writing these two blog posts has also been interesting. I know they ended up very long, and kind of mix technical stuff, CSCL theory, my own ideas about how people work with ideas, and my practical experience in the class. But this is how I think – and it's liberating to not be constrained by the academic paper format, especially in the exploratory phase. Writing it down enables me to reflect more deeply on my own design, and execution. Putting the script into Dillenbourg's orchestration graph format was also a great exercise, even though I have barely scratched the surface of his framework.

The drawing of an internet workflow comes from a student in the P2PU course. The Web 2.0 image comes from Nonprofit Tech for Good. The Do as I tell You sign comes from Ice House Crafts

Supporting idea convergence through pedagogical scripts and Etherpad APIs, an introduction

2014-10-03T20:17:26Z

We can script Etherpad to push discussion prompts out to many small groups, and then pull the information back. Using tags, we can extract information, and as a community organize the emerging folksonomy

This blog post brings together two long-standing interests of mine. The first is how to script web tools to support small-group collaborative learning, and the other is how to support reorganization of ideas across groups.

Two meanings of the word "scripting"

There is an interesting intersection between two quite different meanings of the word scripting in the context of my work. In the CSCL literature, scripts refer to sequences of activities that support groups of students in carrying out a collaborative learning activity. They can be very simple and generic, like the well-known jigsaw script, or very content-specific. There is active on-going research on scripting, including how external scripts interfer with internal scripts, and the dangers of over-scripting.

From the technology side, we talk about writing scripts as a light-weight form of programming, to ask the computer to do something for us. Scripting a program, or a web site, means that we can interact with it automatically, for example asking Google Docs to set up 20 documents for us. There is obviously a connection between developing educational scripts and actually "implementing" them in technology. Often, specialized software has been written to enable specific scripts, but there has been some attempts at enabling more generic implementations of learning designs into software.

One inspiring example for me has been GLUE-PS (paper), which can take a learning design from a learning design software, like CompendiumLD, and set-up the necessary software, for example pre-generating Google Docs and wiki-pages, splitting students into groups, and assigning different resources to different groups, etc.

Etherpad for collaborative writing, and more

For those who have not heard of, or used, Etherpad, I always introduce it as "like Google Docs". It enables very easy collaborative editing of documents, where everyone logged in immediately see all changes made by other users. It features very light-weight formatting, and is not designed to generate print-ready documents, but is perfect for brainstorming, note taking, etc. It also focuses more on the live-editing situation, with the text written by each user colored differently. It can be quite mesmerizing to see a document being filled in by multiple people at the same time.

We often used Etherpad for collaborative meetings at P2PU (case study), and I also had some interesting experiences using in the P2PU course "Intro to Computer-Supported Collaborative Learning". We began by chatting in the chat feature, but gradually migrated over to use the main editing space, which enabled us to keep multiple separate discussions going simultaneously. I have written and spoken about this experience, and it's something I'd love to experiment more with.

For a large course I am currently helping teach, we began by using Etherpads to support note-taking in small group discussions. Setting up 10-15 Google Docs, remembering to set the sharing options correctly, and copying and pasting the URLs to the wiki was a lot of work, and it was much easier to do it with Etherpad. The students all found Etherpad very easy to use, and it was nice to be able to pull up group notes on the projector afterwards while groups were presenting their ideas to the rest of the class.

Scripting Etherpad

Etherpad has a nice simple API, which let's you create new pads and fill them with content, and read the content of existing pads. (Since there are no permissions or logins, that's basically all the actions that you would need). Since Etherpad conserves the full history of all changes, you can also access older versions of a pad. Here is how little code you need to get started.

A simple helper function to execute Etherpad API calls, and properly format the return message:

def run_etherpad(path, **params):
    params['apikey'] = settings.ETHERPAD_API
    data = urlencode(dict(params)).encode('ascii')
    url = ("%s/api/1.2.10/%s" % (settings.ETHERPAD_URL, path))
    r = json.loads(urlopen(url, data).read().decode('utf-8'))
    return(r)

And then I can simply execute any of the API commands listed in the Etherpad documentation, for example:

text = run_etherpad("getText", padID=pad)

The first thing I did with this, was to write a little script that took a template (typically a few questions for discussion), and created a number of identical pre-populated Etherpads for the discussion groups. It then generated an HTML page with all the links, which I could copy and paste into the course wiki. This already saved a lot of time. Given that we now have a list of all the Etherpads, it's just as easy to pull all the text from the Etherpads back into a script, and do something with it.

For example, I can generate a quick HTML page consisting of the text from all of the Etherpads together, which makes it much faster to quickly scan through what groups have been doing, rather than opening 15 tabs in the browser. Or I can automatically pull each group's page and post it to the wikipage of that group for easier access in the future (Etherpad is great for live editing, but not very trustworthy as a long-term repository of information).

Using tags to extract and organize ideas

A little detour about making ideas moveable, supporting organizing and synthesis

One of the things we found from teaching this course last year, is that the ideas and information entered tended to be "stuck" on the week and group-specific pages. After several weeks of the course, we would have lot's of great reflections, notes from readings, project ideas, etc., but they were not organized in a way that was easily accessible to the students. When working on their final projects, the students were more likely to use Google, rather than accessing the common knowledge base which we had tried to structure throughout the course.

I have long been inspired by a course that I took in the first year of my MA, which used an environment called Knowledge Forum. I made a screencast to showcase the course, and how the environment enabled us to go back to our old notes, reorganize them, and see new connections and gaps. At that time, I contrasted it with threaded discussion forums, where a post is usually "captured" in the chronology/thread where it is posted.

Inspired by this course, and by workshop methodology, I began thinking about what I called "the cycle of divergence an convergence" which I began exploring in a talk and a paper. I also thought that tagging could be a low-tech way to enable "movable ideas". An example script could be the following: after a class had discussed a topic for a few weeks, each week with a new "input" (stimulus), they could collectively or individually decide on a few emerging overarching themes, and go back through the posts, tagging them accordingly. The discussion forum software would then create "saved searches" for these tags -- dynamic folders containing all the posts tagged with a certain tag. Students could then revisit these posts in a new context, continue the discussion, etc.

This turns out to be very similar to the process of qualitative research, and a methodology supported by tools like NVivo (good video intro), where one codes individual pieces of text, and then pulls up a list of all the coded pieces for a given code, to see them in a new context. Something that might be useful for quantitative research, for organizing a literature review... and for supporting community knowledge and deep thinking in a collaborative learning situation?

Tag-extract, short recap

Programmers both like to reinvent the wheel, and to scratch their own itch, so faced with a large amount of notes that I needed to structure into a literature review, while keeping the source-information (which paper did a certain idea come from), I experimented with tagging. (Adding some markup to the text, and then processing it with a script, is far easier than developing a new graphical interface for tagging). I had three simple rules, or principles, which turned out to be quite powerful:

all the text below a source indicator "belongs" to that source, and will always be tagged as coming from that source
any tag applies to the line it is on
all text indented below a tagged line, belongs to that line

I wrote a detailed blog-post about this tool, complete with an extensive screencast, showing the tool embedded in an academic workflow process, and a more focused screencast focusing on the tool itself.

At the top, you see a text file with notes, and bibdesk-identifiers as the "source". After adding tags, and running the script, I get the output seen on the right, where the notes are reordered according to tag, but with the source still there. When writing about for example metalearning, you then have all the relevant ideas/quotes, together with the article citations. Here's an example of raw sorted notes about open courses, and here the resulting draft.

Bringing all the pieces together

So we have introduced the idea of pedagogical scripting, as well as implementing scripts in computer code. I've discussed the desire to make ideas more "moveable", and support deeper work on ideas, and talked about the idea of using tags to support this. Finally, I introduced the tool tag-extract, which I developed to work on a literature review. This blog post is already long enough, so I will write a separate blog post about the actual design, implementation and evaluation of the pedagogical script using these ideas.

The script flowchart is from an unpublished manuscript by Pierre Dillenbourg. The GLUE-PS image is from the GLUE-PS website, the concept map about concept maps from the Cmap site, and the NVivo image from the NVivo primer

Easy interoperability between Ruby and Python scripts with JSON

2014-09-22T21:57:15Z

I recently needed to call a Ruby script from Python to do some data processing. I was generating some Etherpad-scripts in Python, and needed to restructure tags (using tag-extract) with a Ruby script. The complication was that this script does not just return a simple string or number, but a somewhat complex data structure, that I needed to process further in the Python script.

Luckily, searching online I came across the idea to use JSON as the interchange format, which worked swimmingly. Given that all the information was in text format, and the data structure was not that complex (just some lists and dictionaries), JSON could cope well with the complexity, and was easier to debug, since it's a text format. If I had had other requirements, like binary data, I would have had to investigate other data formats.

The Ruby script already had several different output options, controlled through command-line switches. I added the option to output through JSON, and the code required to do the actual output was simply:

puts JSON.generate(tags)

I also reconfigured the script to accept input from standard-in:

if ARGV.size == 0
  a = ARGF.read
else
  a = try { File.read(ARGV[1]) }
  unless a
    puts "Could not read input file"
    exit
  end
end

This meant that I could simply pipe the information from the Python script through the Ruby script, writing to standard-out and reading from standard-in, not having to create any temporary files at all.

I created a helper function in Python to let me run a command, putting into standard-out, and reading from standard-in:

def apply_external(cmd_ary, out):
    proc = subprocess.Popen(
        cmd_ary,stdout=subprocess.PIPE,
        stdin=subprocess.PIPE)
    proc.stdin.write(bytes(out, "UTF-8"))
    proc.stdin.close()
    result = proc.stdout.read()
    result = result.decode("UTF-8")
    return(result)

and then I simply call it with the text I want transformed, and receive the transformed text structure back.

def tag_extract(text):
    ret = apply_external(['ruby', '/Users/Stian/src/folders2web/tag-extract.rb'], text)
    return json.loads(ret)

Although the idea was simple, there was a bit of fiddling getting the pipes set up in Python, etc., so I figured that documenting it might be helpful to others.

Creating Anki cards for Russian Coursera MOOC with stemming and frequency lists

2014-04-16T18:44:51Z

In which I automatically generate Anki review cards for vocabulary based on subtitles from a Russian Coursera MOOC

Learning Russian, a 10-year project

I have been working on my Russian on and off for many years, I'm at the level where I don't feel the need for textbooks, but my understanding is not quite good enough for authentic media yet. I've experimented with readings novels in parallel (English and Russian side by side, or Swedish and Russian, like on the picture), and I've been listening to a great podcast for Russian learners.

Language learning with MOOCs

MOOCs can be a great resource for language learning, whether intentionally or not. At the University of Toronto, we found that more than 60% of learners across all MOOCs spoke English as a second language, no doubt some of them are not just viewing the foreign language of the MOOC as a barrier, but are also hoping to improve their English. To my delight, the availability of non-English language MOOCs has been growing steadily. For example, Coursera has courses in Chinese, French, Spanish, Russian, Turkish, German, Hebrew and Arabic. There are also MOOC providers focused on specific linguistic areas, for example China's XueTangX, France's Université Numérique, and Germany's iversity.

The existence of global open educational resources is nothing new. Five years ago, I wanted to share my enthusiasm about being able to "peek through the windows" of universities around the world, and edited a YouTube mashup. I've also blogged about OER for a multicultural classroom, and written about OER from China, India, Indonesia, etc.

However, earlier OER videos tended to be full lecture recordings, with poor video and audio quality, often on proprietary and hard to use platforms, with very little facilitation for online learning. Current MOOC platforms have converged around short (5-15 minutes) videos, with high video and audio quality, and decent video players. The courses are coherent, designed for online learning, and often feature supporting readings, sometimes even subtitles, etc.

A Russian MOOC

The picture above is from a lecture in the MOOC "История экономической мысли" (History of Economic Thought), which is currently in it's second week on Coursera. I signed up for this several months ago, because I wanted to try out my Russian, and I find the topic very interesting. (Extra interesting, of course, is the question of whether this topic will be taught any differently from a country that has a very unique history with regard to economic thought and experiments). The fact that I know something about the topic, and about European history in general, makes it easier to follow the lectures, as do the subtitles, the ability to regulate the speed, and the occasional illustrations in the video. Watching a few of the videos, I found that I could more or less follow along, but there were a number of words that seemed important, and which I could not understand.

Stemming and word frequencies

I've always been interested in language technology, and last year I spent some time experimenting with different libraries to create word frequency lists from Russian electronic texts. I used a library to "stem words" -- reducing all the conjugated forms of a word to a single word -- for example all these forms of the word big: большая, больше, большей, большие, большим, большими, большое, большой, большую. There are several possible libraries for doing this, none are perfect, but most are quite good. In my previous blog post, I used the Ruby gem ruby-stemmer, however when I revisited my code, I noticed that I had switched to using hunspell after writing the blog post.

Hunspell is an open-source spellchecker, which is also able to output word stems. In the example below, I simply send a Russian sentence through hunspell, and get back a list with the original conjugated word on the left, and the stem on the right (I've removed the words that didn't change)

❯❯❯ echo "Сегодня наше предствавление об экономике очень сильно отличается от того,
    как люди древности эээ, мыслили то, что мы сегодня называем экономикой." |
    hunspell -d ru_RU,ru_RU_google -s

экономике экономика
сильно сильный
отличается отличаться
древности древность
мыслили мыслить
называем называть
экономикой экономика

Getting word frequencies from a MOOC subtitle file

Not only did I have this old project with stemming and creating frequency lists of Russian words, but I have recently done some experiments with downloading video subtitles from Coursera courses for use in a machine learning project. This script takes an HTML file with the list of course lectures (you need to be logged in to get to this page, so the easiest is to open it in the browser, and save the page), extracts all the subtitle links, downloads them, and concatenates them to one output file.

We can then pass this file through the stemmer (all code for this blogpost on Github), and generate a list of all word-stems that occur more than five times, sorted by general frequency. (I use a Russian frequency word list I found here). The subtitles for the first two weeks of videos (18 videos) total 27,600 words (50-80 pages of printed text). There are almost 6,400 unique words (of which сельскохозяйственного -- agricultural -- is the longest), but only 3374 stems. By removing all stems that occur less than five times, we're down to a much more manageable 839 stems.

I opened these in a text editor, and manually removed words I already knew (I left in many words which I kind of knew, but would like to learn more precisely). Since the words were sorted in order of global frequency, with more "rare" words near the top, I could remove most of the lower half of the file in one swoop (words like "I", "like", "and", etc). Within half an hour, I had now gone from 27,600 words to 184 word stems that I needed to learn or study.

No problem, just copy the whole list of Russian words into Google Translate, and out comes an equally long list of English corresponding words, which I save to another text file. (These are not 100% perfect, but in my experience, it works for 95% of the words).

Bringing it all together - Anki and spaced repetition

A common concept on language learning blogs is "spaced repetition", the concept of having a computer automatically determine the optimal repetition rate to counteract the natural rate of forgetting. Anki is a great free program for practicing flashcards using spaced repetition, with very powerful options to create your own cards, configuring the rate of repetition, etc. I have only ever used it before with public decks, but I thought I would try to use the tools I introduced above to make my own cards.

The simplest cards can be represented as two columns in a file, the front and the back. You are presented with the word on the front, try to remember the word on the back, turn the card, and indicate whether you were successful or not. However, Anki supports much more powerful card configurations. You create a new note type, with arbitrary many fields, and then you can create different card designs, incorporating these designs.

I decided that it would be very helpful to see the words in context as well. And not just in an arbitrary context, but in the specific context from the course. Given that I had already gone from specific words to the word stem, I could go back to using those specific words to search for example sentences, add highlighting of the word in question, and include these in the file. Each card has a Russian word on the front, and when you turn it, you see the English translation (from Google Translate), the Russian stem (for reference), and up to three example sentences with the word highlighted.

I think the result is pretty awesome. It will be fun to practice on these cards, and then go back to watching the videos. When next week's videos are released, I should be able to quickly generate a list of words that are new (were not covered in week 1 or 2).

There are of course many ways in which this could be improved. I'm thinking of how to keep track of words I know, so that I can easily remove them when getting a word list from new material. Integrating this kind of functionality into the Coursera or other MOOC platforms would be neat, although it will probably never happen. (If they had more open APIs, other people could build it, however). I could for example easily extract the timing of the example sentences, and include that in the flash cards, so you could jump straight to a specific video at the point where the professor says a certain example sentence (however, I don't know how to generate URLs that open a Coursera video at a specific time).

There's also a lot of other fun stuff one could do with subtitles - it would be great if we could get easy access to subtitles for all the courses, and use it to build a search engine (linked directly to videos etc.)

But for now, I need to study some flashcards, and watch some Russian videos. Пока!

(All the code on Github)

Learning idiomatic Haskell with Exercism.io

2014-03-11T15:50:24Z

I got my introduction to functional programming through Clojure, but lately I've been really fascinated by Haskell. I gave up on it a few times, because it can seem quite impenetrable, and very different from anything that I'm used to. But there is something about the elegance, and powerful ideas that keeps me coming back. I've read a bunch of tutorials and papers, followed blog posts, and experimented a bit with ghci and IHaskell, but I never really got off the starting block with writing my own Haskell.

How to get practice?

The projects I'm currently working on in Python are too complex and urgent for me to try to implement them in Haskell at this point, and because Haskell is so different, I found myself stymied doing even simple things, even though I'd just finished reading a complex CS paper about monads. I needed some structured tasks that were not too hard, and that gave me good feedback on how to improve.

Listening to the Functional Geekery podcast, I heard about exercism.io, which provides exercises for many popular languages. There are several similar websites in existence, perhaps the most well known is Project Euler, however that focuses too much on CS/math-type problems, and is perhaps better for learning algorithms than a particular language. The way exercism.io works, is that you install it as a command line application. The first time you run it, it downloads the first exercise for a number of languages.

The exercises themselves consist of test cases, and it's up to you to write a module that will fulfill the test case. Once you are done, you use the command line client to quickly submit the answer, which unlocks everyone else's answers, and downloads the next exercise. The fact that you download the test cases and README files, and work on your local computer, is also great. You can use the environment you're used to (Emacs, VIM, EclipseFP), with all the tooling, documentation, autorunning tests, etc., that you are used to.

Trying before being told

This kind of graded approach is brilliant, because it precludes "cheating". It's easy to skip ahead to the solution, say "that makes sense, I would have come up with that", and move on. But it's important to actually force yourself to come up with that. Reading other people's versions, and the feedback they received, after having struggled with the problem yourself, is also incredibly informative -- and again, I think you get far more out of it, than just browsing through solutions before you've put in the hard work yourself.

There's a parallel to some of the work I'm doing with audience response systems in lectures. Eric Mazur at Harvard puts physics problems on the projector, and asks students to first try to solve them themselves. Then they have to try to convince the person next to them about the right answer, which they submit by clicker. Finally, he explains and discusses. Students who have wrestled with the problem for a while already, and been exposed to other people's ways of thinking, are far more receptive and learn more from the final explanation, than if you had just gone directly to lecture.

Idiomatic code

The objective is not just to pass the tests (which is in most cases trivial, for example it would not be difficult to write very specific code that just fulfills the test cases, but does not generalize), but to write as beautiful, idiomatic code as possible. One might start with an inelegant approach that fulfills the test cases, and then begin refactoring into something one can be proud of, while making sure the test-cases continue to work.

This is an interesting challenge, and it's also very useful, because one of the biggest challenges when learning a new language is internalizing the metaphors and idioms of that language. When writing a simple function, there might be many ways of writing it, that will all "work", and yield the same result. Yet once you begin writing larger and more complex programs, programs that need certain kinds of performance (memory, CPU, dealing with very large inputs etc), the difference between different approaches becomes very large.

The first project I did was quite embarassing. The task asks you to construct a function that takes a string, and generates a string as a response. There are a bunch of test cases, and you have to figure out yourself how to group them (seems like strings ending with ? are questions, but not if they are all upper-case, etc). I decided to use pattern matching, which was a fair approach.

However, I instinctively used regexp patterns to match the text, instead of looking for built-in functions. I got it to work mostly (and did learn how to use regexp in Haskell, which I hadn't used before), but I had problems with unicode, and spent too long trying different patterns. After I submitted, I saw an incredibly elegant solution, using only built-in predicates like isAlphaNum, which read and compose much better than my convoluted regexps. A very good lesson, and one I internalized much quicker than if I read a blog post about it, without having struggled with it myself first.

Future growth of exercism.io

The platform is quite young, and I'm wondering how it will scale. For the first few exercises in Haskell, there were only about 15 responses (since you cannot proceed until you have submitted the first few responses, I'm guessing the more advanced exercises will have even fewer answers). If there were hundred answers for a given problem, I might want to see as many different approaches as possible, not 20 examples that are basically the same (I'm also interested in as much diverse feedback as possible).

Perhaps some kind of voting feature is necessary. There is the possibility of providing feedback to individual submissions, and while I really appreciate people's feedback, I almost feel like it isn't necessary once I've seen other people's contributions -- I already have a good idea of what I could do better. (There are only so many different ways of doing these fairly small and bounded problems).

Anyway, I was very excited to discover this platform. After the first task, which I spent about two hours on, I did two more. With these, I was less embarassed when seeing other people's code, but in both cases, there were more elegant solutions than the one I suggested. One of the recurring themes also seems to be really understanding the core library, and all the functions that are already implemented, instead of trying to reinvent the wheel.

Go learn some code!

Parsing massive clicklogs, an approach to parallel Python

2014-03-10T15:09:17Z

I am currently working on analyzing MOOC clicklog data for a research project. The clicklogs themselves are huge text files (from 3GB to 20GB in size), where each line represents one "event" (a mouse click, pausing a video, submitting a quiz, etc). These events are represented as a (sometimes nested) JSON structure, which doesn't really have a clear schema.

Introduction

Our goal was to run frequent-sequence analyses over these clicklogs, but to do that, we needed to process them in two rounds. Initially, we walk throug the log-file, and convert each line (JSON blob) into a row in a Pandas table, which we store in a HDF5 file using pytables (see also). We convert the JSON key-values to columns, extract information from the URL (for example /view/quiz?quiz_id=3 results in the column action receiving the value /view/quiz, and the column quiz_id, the value 3. We also do a bit of cleaning up of values, throw out some of the columns that are not useful, etc.

Speeding it up

We use Python for the processing, and even with a JSON parser written in C, this process is really slow. An obvious way of speeding it up would be to parallelize the code, taking advantage of the eight-cores on the server, rather than only maxing out a single core. I did not have much experience with parallel programming in general, or in Python, so this was a learning experience for me. I by no means consider myself an expert, and I might have missed some obvious things, but I still thought what we came up with might be useful to others.

There are many approaches to parallelizing code, and choosing the right one requires thinking about your algorithms -- which functions need access to shared state (and can you design it so that you minimize shared state), are you CPU-bound, or IO-bound, what is the grain size that it makes sense to split at, etc.

Shared state

I was using HDF5 as my storage format, and pytables can append to an existing table. Theoretically, one would think that the lines in the log file are completely independent from each other, just get them converted and into the HDF5, and once you've got them all entered (appended), you can index and sort by timestamp, username, etc. However, Pandas Data.Frames have one big limitation compared to R data.frames -- no categorical column type. In R, a data.frame might have a column that stores the answer to a likert-type question (Not at all to Totally agree, for example), and even though there are 10,000 rows, those five text answers are only stored once, with each cell only containing a pointer to the correct alternative. This makes the table take much less space.

Pandas was developed for financial analysis, and still seems fairly focused on numbers. (It also builds upon NumPy, which also doesn't have a categorical format). In our case, this was a big problem, since the difference between storing "/view/lecture" (13 bytes) and 1 (1 byte) for a table with 14 million rows is huge! (Especially when you consider that Pandas does much of its work by loading the entire table into memory).

I decided to hack my own little categorical format, by creating a memoize function, which would assign a unique number to each new value it was called with, but always return the same number for the same value. Quick example:

memoize("hello") #       => 1
memoize("how are you") # => 2
memoize("hello") #       => 1
memoize("how are you") # => 2
memoize("i'm fine") #    => 3

Initially the function simply kept state in a Pandas Series, and once I had finished parsing the entire clicklog, I wrote these series to the same HDF5 file (HDF5 files can contain multiple tables, files etc). This worked great, and it meant that instead of this:

{"key": "user.video.lecture.action","value": "{\"currentTime\":257.066689,
\"playbackRate\":1,\"paused\":false,\"error\":null,\"networkState\":1,
\"readyState\":4,\"eventTimestamp\":13752243249891,\"initTimestamp\":
137524554345511,\"type\":\"ratechange\",\"prevTime\":227.066689}","username":
"9930a0523403499b7347707c92bccbcbff1a","timestamp": 13752234234277,"page_url":
"https://class.mooc-platform.org/course-001/lecture/view?lecture_id=31","client":
"spark","session":"30423409-1323423042304","language": "en-US,en;q=0.5","from":
"https://class.mooc-platform.org/course-001/lecture/view?lecture_id=31",
"user_ip": "123.255.122.167","user_agent": "Mozilla/5.0 (Windows NT 6.1; rv:22.0)
Gecko/20100101 Firefox/22.0", "12": ["{\"height\":678,\"width\":1207}"], "13": [0]}

...I ended up with what was externally represented as a list of 18 floats (I wanted to store them as integers instead, but Pandas needs floats to be able to represent NaNs, which was important, given that many of the values were sparse).

The problem with shared state

This worked well before I tried to do any kind of speedup (apart from the fact that the script took 12 hours to run on the largest file). However, I wanted to be able to iterate a little bit quicker, and explore ways of speeding up the code. Because I needed to keep the memoized data consistent, I couldn't just fork off a bunch of processes that each would process their own data -- they would have all ended up with their own individual memo tables, where 1 corresponds to hello in one process, and what's up in another. To address this, I moved the memoization code out of the function that processed individual lines, and applied it to each Pandas column in a final step before appending the processed chunks to the HDF5 file.

The function to process each line should now be a pure function -- that is, a function that always returns the same output given the same input, not reliant on any external state, etc. -- and thus ripe for parallization. I tried using pool.map from the multiprocessing library to let the computer automatically figure out how many external processes to run, etc. Here I ran into a problem of grain size... It takes a bit of time to fire up processes, transfer state, coordinate across threads etc, so if you assign each process too little work, the gain from having work done in parallel is more than lost. This was the result -- my naive attempt led to a slowdown. (Some libraries try to tackle this automatically and figure out the correct grain size, but pool.map didn't seem to be able to).

My second attempt was to do the chunking myself -- I read in 100,000 lines, chunked it into chunks of 20,000 lines, and then mapped over this list of lists. This worked better, but I only saw a small speedup (maybe 50%). Partly because of the extra time spent chunking lines, and partly because a big part of the program was still spent converting the list of dicts to a Pandas DataFrame, memoizing the columns, and writing to file (I could see this very clearly by monitoring top -- first you see five processes maxing out their respective cores, and then suddenly they drop away, and there is only one process on one core working).

Map-reduce

A very common paradigm for cluster computing (with multiple computers) is called map-reduce. Basically, you split the data into chunks, run parallel operations on each chunk, and then somehow "reduce" the result back together. An example could be getting number of page views per URL for an extremely large clicklog, have each serve count up clicks per URL for a piece of the log file, and then have a script that combines the numbers in the end.

Something similar can be done on a single computer, and for the second phase of the analysis, this is what I ended up with. The second phase, after we got the nice HDF5 file, is to do some processing for each user. Instead of the fact that a user watched video 39, we want to know if he/she watched the video for the first time, or if it was an immediate review, to be able to compare patterns across students, and even across courses.

This means that we cannot treat each student event individually -- we need to know which videos the students has already seen, etc. However, we can treat each student individually -- how one student's actions are tagged, are not at all reliant on other students' actions.

Split, process, collect

One very nice thing about HDF5 is that once you've indexed the table (which is quite fast), you can read in data based on simple queries. Thus, once we've opened the HDF file with store = pd.HDFStore("hdfstore.h5"), we don't have to read the entire table into memory with db = store['db'], which might take 6GB of memory (with a big table), but we can instead run

user = store.select('db', pd.Term('username = 3434')

(in our case, we also select on timestamp, because we only want events that happened while the course was active). Add to this that pytables is thread-safe for reading (ie. multiple processes can read from the same file, without problems), but not for writing (it's not like a mysql database that can have multiple processes writing at the same time).

So instead of having one script that tries to parallelize subtasks, what I ended up doing was to run eight Python scripts concurrently. I simply indicate as a command parameter which usernumber to start with, and how far to jump each time. Thus, the first Python process starts with user 1, then does user 9, user 17 etc, until there are no more users. The second process starts with user number 2, then 10, etc. This way all users get processes, without the need for any coordination. Initially I started the processes from a shell script, but then I put it into Python:

print("Spawning %d processing scripts" % numprocesses)
procs = []
for proc in range(1, numprocesses+1):
    range_start = proc
    range_jump = numprocesses+1
    procs.append(Popen([python_exec, "videos_reduce.py", hdffile, str(range_start), \
        str(range_jump), tmpdir, cutoff]))

I can then wait for all processes to finish, and then do any cleanup work that is necessary.

def all_procs_finished(procs):
    for proc in procs:
        if proc.poll() == None:
            return False
        return True

while not all_procs_finished(procs):
    time.sleep(5)

Reduce

Initially, I wanted to capture the output in a new HDF5 file, so I had the individual scripts dump their output pickled into temporary files (named using uuid) in a temporary directory, which would then be read by the main script, and appended to the HDF5 file (thus ensuring that only one thread wrote to the HDF5, to avoid data corruption). Later, we decided we wanted to output a text file for processing in R, which made it even easier. The individual thread simply spit out a bunch of text file chunks into a temporary directory, and then we combine them with cat * > outfile.

This approach worked swimmingly. Since we are just running the same script eight times in parallel, it's easy to debug by running the script only once (I would run it with 1 10000 to start with user 1 and jump 10000 ahead, to 10001, 20001 etc until the file had been exhausted). Because all the hard work was done in the individual threads, I got very close to 100% utilization on all eight cores for the duration of the run (which is a beauty to see in htop), and even when running a separate process that collected the output files and wrote them to HDF5, that process used so little CPU as to be negligible.

But what about that shared state?

At this point, I needed to go back to the initial phase to redo a bunch of the parsing, because I had received new files. Inspired by my experience with the second phase, I thought about how I could improve the first-phase scripts. The tricky problem was how to coordinate state. I did a bunch of googling to look at "coordinated / parallel memoization libraries for Python", and somehow came upon people using Redis for that purpose.

I had heard very vaguely about Redis, but never known much about it. It turned out to be perfect for the job through. It's basically a very fast distributed key-value store (although it also has other value types, like lists and hashes), with an API that only takes a few hours to learn. I installed a Redis server, and rewrote my memoization function to use Redis instead of the Pandas Series, with only very minimal changes.

Here is a shortened version of the memoization class:

class Memo(object):
    def __init__(self, name, prefix):
        self.r = redis.StrictRedis(host='localhost', decode_responses=True)
        self.name = name
        self.prefix = prefix
        self.counter = '%s:%s:cnt' % (prefix, name)
        self.store = '%s:%s:store' % (prefix, name)
        if self.r.zcard(self.store) == 0:
            self.add_object('None')

        def add_object(self, obj):
            c = self.r.incr(self.counter)
            self.r.zadd(self.store, c, obj)
            return c

        @lru_cache(maxsize=256)
        def get(self, s):
            if s is None or pd.isnull(s):
                return -1

                existing_score = self.r.zscore(self.store, s)
                if existing_score:
                    return existing_score
                else:
                    c = self.add_object(s)
                    return c

You create a new Memo class with a name, and then just call it like described above. lru_cache, from functools, is really neat - it remembers the last n (maxsize) arguments to the function, and their return values, and if you call the function with an argument in the list, it gives you the response, without having to run the function. This is safe, given that once a value is assigned in Redis, it will never change. Having this meant a big speedup (especially since it's very common for adjacent lines to have similar values, for example a chunk of lines all from the same user, etc), however when I tried to set maxsize=None, ie. to remember all the values, it became much slower. Apparently Redis is much faster at looking up a hash-value among 10,000 than Python, despite the overhead of calling out.

So the final version of the script does several thing. It begins by running the shell command split -n, which splits the log file into chunks based on number of lines (currently 40,000 lines seems to be a good size). It does not wait until cut is finished, but begins immediately launching the 8 processes, which begin reading the next available file in the temp dir, and processing it.

The eight workers pick up a temp file "chew on it", and spit out the result as a pickled array of dicts in another temp dir. The master script, having launched all the processes, begins to cycle, waiting for files to appear, reads them in, and appends them to the HDF5 file. The worker processes continue until they've gotten the signal that cut has finished (through Redis), and there are no more files. The master script continues until all worker processes have finished, and there are no more "chewed" files. It then repacks the HDF5 file (to add indices), and adds the memoized data from Redis.

Using this approach, I am now able to process a 20GB clicklog file in 12 minutes! (Resulting in a 3.5 GB HDF5 file with something like 14 million events).