In this paper we look at how a web-based social software can be used to make qualitative data analysis of online peer-to-peer learning experiences. Specifically, we propose to use Cohere, a web-based social sense-making tool, to observe, track, annotate and visualize discussion group activities in online courses. We define a specific methodology for data observation and structuring, and present results of the analysis of peer interactions conducted in discussion forum in a real case study of a P2PU course. Finally we discuss how network visualization and analysis can be used to gather a better understanding of the peer-to-peer learning experience. To do so, we provide preliminary insights on the social, dialogical and conceptual connections that have been generated within one online discussion group. p. 1

How do we capture observations relating to learning experiences which unfold in a virtual space and are mediated by specific technologies? p. 2

In this paper we propose a different use of Cohere: as research tool for assisting the qualitative analysis of online learning experiences. In section one we discuss the benefits for using Computer Assisted Qualitative Data Analysis (CAQDA) tools for virtual ethnography and online observations. in Section two we describe Cohere and how it support the main QDA functions of coding and memoing. We continue section three with a description of initial results of a case study in which we explored how peer-to-peer learning takes place in an online course. Although we do not seek to generalize from observations, insights from visualization of data analysis are used here as a ‘proof of concept’ of the ways in which Cohere can be used for CAQDA in virtual learning environments. Finally in section five we discuss lessons learned and future work. p. 2

Traditional qualitative analysis tools do not allow web data to be analyzed "in-situ", but must be downloaded, normalized etc. p. 4

More importantly, when data are saved to be analyzed they are disconnect from the virtual context (Web pages) from which they have been extracted. This generates two level issues:

• The analyst looses potentially useful hints that may help him to make sense of the content they analyze (i.e. page formatting, images and metadata visible just when the data is accessed online)
• If new data are added to the data source Webpage they are not visible to the analyst. Therefore the results of the analysis are time-constrained and may become quickly obsolete. p. 4

Cohere provides two of the main features for QDA: Coding and Memoing. Coding and Memoing activities of qualitative Web data can be assimilated to the common users activities that in the Social Web ‘language’ are defined as tagging and annotation. By providing collaborative tagging and web annotations Cohere enables qualitative data analysis in a web environment. p. 4

Coding as Tagging and Memoing as Web Annotation p. 4

Use Case: Observing Virtual Users in a Peerto-Peer You University (P2PU) Course p. 5

Peer-to-Peer University (http://www.p2pu.org) has evolved into a public space that demonstrates the fostering of collaboration among activists within the open education movement, volunteer tutors and motivated learners in the design and facilitation of short courses. p. 5

The ‘Copyright for Educators’ course during its second cycle (March-May 2010, http://www.p2pu.org/copyright-educators-cycle-2-mar-2010) is the object of analysis for this paper. p. 6

The object of analysis are the discussion forums, where course students collaborate in order to complete group assignments or discuss given tasks. In particular we analyzed students’ posts in the “Pink Group” (http://p2pu.org/node/729/document/2692). p. 6

we focus coding on the following two interrelated dimensions, based on Burge’s (1994) peer bahaviour models: a) participation: how do participants give alternative perspectives attending to the experience of others: how do they share resources and reflections? b) Affective feedback: do participants use each others’ names, complimenting each other and offer supportive, remedial or critical interchanges? p. 6

Firstly we applied coding and memoing. We tagged clips of forum discussion’s text by keeping in mind the general question: What is relevant here for the phenomena I want to observe? We tried to identify, name, categorize and describe the phenomena found in the text. We looked at each post, highlighted clips of text, and tag them with codes, which quickly started grouping in three main categories:

• People
• content
• Rhetorical moves p. 6

We identified two main memos connection codes:

• Posts
• Addressed to p. 6

Another key activity in QDA is sorting. p. 7

Cohere has a network approach to data sorting and visualization. Data, codes, and memos can either be listed (ordered by creation time or code type) or represented in a graph like structure. p. 7

Figure 4 for example, shows the representation of the activities of the Pink group as they have been observed and coded in the use case. By looking at the image we can recognize that 4 participants have been addressing the group attention on different aspect of the learning experience. Some of them, for instance, by asking questions to the entire group (see question icons in Figure 4). p. 7

Results in Fig.5 show that there are 6 cluster, that is to say 6 particiapnts in the discussion forum and it is equally evident who are the most active and what activities they have carried out. This visualization offers a useful way in for data exploration and reflection. It is easy for the analyst to familiarize with the data and make visual and conceptual complarison between the emerging codes and their relationships. p. 7

Finally network search can be coupled with code search in order to spot more specific relationship between data. By searching for the code “complain”, for example, and by focusing p. 7

on posts addressed to the all group we can see what participants have declared to be uncomfortable with the technology or unhappy with the course organization, and what participants have posted more complains (Fig. 6). p. 8

Even thought this particular P2PU case study is still in progress we can describe some lesson learned on the course we analyzed, in particular on course pedagogical design. Although there is no cohesive design across the courses, the most popular activities that seek to facilitate the mediation of what we would call a ‘learning space’ follow a similar pattern for engaging peer and collaborative learning: ‘read-think-reflect-share-peer comment’. Observations within the site (and metapages, including discussions with facilitators) suggest that ‘Copyright for Educators’ has implicit pedagogical designs that evolve during the 6 weeks that the course runs. p. 8

The figure below exemplifies pedagogical design implicitly evolving within this particular learning space. as a process. The intended outcomes and audience are explicitly stated in the course outline, and the pool of learning materials is structured in such a way to promote an inscribed pedagogy that addresses three core dimensions (see bottom left corner – part of tutors’ role in relation to defining learning outcomes): i. Informative: help indentifying copyright issues around education ii. Practical: work with different IP jurisdictions and the legal practicalities for open education design iii. Social and deliberative: exchange ideas about open education beyond and within the context of copyright The network of interactions mediated through the interface tools facilitated by the site (see roles, tasks and activities in the map), produces a set of novel resources that if visualized p. 8

appropriately, it presents structures of intended learning, and the ‘learning in use’. These can guide or inform the design process for future outlets within this particular open course, or other OER(s). p. 9

Bibliographic references p. 13

Anna De Liddo is Research Associate at Knowledge Media Institute of the Open University (Milton Keynes, UK), where she works in the Open Learning Network project (www.olnet.org) at the design and development of a Collective Intelligence socio-technical infrastructure to enhance collaborative learning in Open Education. She gained her PhD at Polytechnic of Bari, Italy, investigating ICT for Participatory Planning and Deliberation. After that, she held a PostDoc position at the Open University within the ESSENCE Project, investigating and evaluating human-centred computing tools to help tackling wicked problems such as Climate Change. p. 13

Panagiota Alevizou is a post-doctoral researcher at the Open University's Institute of Educational Technology working on the Open Learning Networks project (Olnet). As part of her research she has been engaging with numerous stakeholders and projects in the OER community to develop working framework on the nature of openess and collaboration that characterizes the mediation of open resources, while addressing the opportunities and challenges relating to participatory interfaces, web 2.0 pedagogies, adoption and (re)use. Her broader background is in media and communications and her interest in open source/ open content communities was sparked while working on her PhD and has informed her teaching and research at the universities of Sussex and London School of Economics and Political Science p. 14