We propose the concept of Contested Collective Intelligence (CCI) as a distinctive subset of the broader Collective Intelligence design space. p. 2

Abstract. We propose the concept of Contested Collective Intelligence (CCI) as a distinctive subset of the broader Collective Intelligence design space. CCI is relevant to the many organizational contexts in which it is important to work with contested knowledge, for instance, due to different intellectual traditions, competing organizational objectives, information overload or ambiguous environmental signals. The CCI challenge is to design sociotechnical infrastructures to augment such organizational capability. Since documents are often the starting points for contested discourse, and discourse markers provide a powerful cue to the presence of claims, contrasting ideas and argumentation, discourse and rhetoric provide an annotation focus in our approach to CCI. Research in sensemaking, computer-­‐supported discourse and rhetorical text analysis motivate a conceptual framework for the combined human and machine annotation of texts with this specific focus. This conception is explored through two tools: a social-­‐semantic web application for human annotation and knowledge mapping (Cohere), plus the discourse analysis component in a textual analysis software tool (Xerox Incremental Parser: XIP). As a step towards an integrated platform, we report a case study in which a document corpus underwent independent human and machine analysis, providing quantitative and qualitative insight into their respective contributions. A promising finding is that significant contributions were signalled by authors via explicit rhetorical moves, which both human analysts and XIP could readily identify. Since working with contested knowledge is at the heart of CCI, the evidence that automatic detection of contrasting ideas in texts is possible through rhetorical discourse analysis is progress towards the effective use of automatic discourse analysis in the CCI framework. p. 2

CCI is conceived as an emergent capability which depends on being able to pool and connect people’s interpretations, comments and debates around issues, often anchored in diverse documents p. 3

CCI is relevant to the many organizational contexts in which it is important to work with contested knowledge, for instance, due to different intellectual traditions, competing organizational objectives, information overload or ambiguous environmental signals. The CCI challenge is to design sociotechnical infrastructures to augment such organizational capability. To cope with such dilemmas, we will argue that organizations need ways to construct plausible, possibly competing, narratives. This motivates the design of a CCI platform that mediates, captures and structures contributions that may be in tension. For this reason, we see discourse, signaled in texts by distinctive rhetorical moves, as providing a CCI infrastructure with important cues to relevant phenomena. p. 3

This paper extends the initial CCI proposal by De Liddo and Buckingham Shum (2010) with a more detailed presentation of the concept, a conceptual framework attending specifically to the role of document annotation, the addition of automated annotation technology, and a case study evaluating combined human/machine corpus analysis. p. 3

In well-­‐understood problem spaces, there is an objectively optimal or correct response, even if this is only apparent in hindsight. In many other contexts, however, such as horizon scanning, intelligence analysis or public policy formulation, there will almost always be contention over the right answer, and indeed, over how to frame the problem and success criteria (Rittel and Webber 1973). In organizational life, different intellectual traditions (both academic and professional), or competing organizational objectives (e.g. different teams with divergent priorities) invariably set up debates of different sorts. p. 4

Sensemaking has emerged as a definable research field over the last 30 years, dating back to Doug Engelbart’s visionary 1960s work on the need for tools to “augment human intellect” in tackling “complex, urgent problems”, Horst Rittel’s formative work in the 1970s on “wicked problems” (see Buckingham Shum 2003, for a review), and Brenda Dervin’s work within communication studies (Dervin and Naumer 2009) p. 4

journal issue devoted to the subject (Pirolli and Russell 2008), influential work has also “emerged quasi-­‐independently in the fields of human-­‐computer interaction (Russell et al. 1993), organizational science (Weick 1995), and cognitive science (Klein et al. 2006).” See Selvin (2011, forthcoming) for a detailed review of the sensemaking literature from a human-­‐centred computing perspective. p. 5

Extensive research into the interplay of cognition and external representations (e.g. Scaife and Rogers 1996), confirms that the very process of externalizing thought shapes unfolding understanding, from attempting to verbalize inchoate thoughts, to sketching diagrams, to codification in structured symbol systems. p. 5

The CCI challenge may thus be framed as one of creating infrastructures capable of gathering, externalizing and socially validating accounts about past, present and future worlds, in order to establish “plausibility, coherence, and reasonableness”. Such accounts are narrative p. 5

We envisage a future CCI platform capable of providing insight into phenomena such as the intellectual structure of an emerging topic, the quality of the online discourse, how the social network relates to recommended courses of action, how stakeholders are framing problems, what the claimed gaps in understanding are, the assumptions being questioned, and the diverse forms of reasoning being deployed (e.g. technical, commercial, political, ethical). p. 6

to operationalize meaningfully the concepts and processes central to such narratives in order to make them more computationally tractable, but also, to ensure that these remain amenable to human inspection and reasoning, to enable continued social negotiation over the legitimacy and significance of artifacts in the system. p. 6

The evidence from systems that seek to automate reasoning through knowledge representation techniques is that analysts reject ‘black boxes’ they cannot interrogate, preferring information management tools with intuitive visualizations that leave them in control of higher order reasoning and judgments about the significance of a data point or argument (Lowrance et al. 2008). A CCI platform should make transparent why, for instance, it represents two documents as being in a contrasting relationship, or why the collective view seems to be that the evidence for one course of action is strong. p. 6

discourse. p. 6

Hagel et al. (2010) point out that much of the relevant knowledge in novel, emergent domains and social systems has not yet been Contested Collective Intelligence 6 formally codified p. 6

The importance of discourse p. 7

formally codified p. 7

Once a set of competing viewpoints has been identified within a CCI platform, how can these be analysed? It is not surprising to find relevant work within the intelligence analysis research community. We already referred to Lowrance et al.’s (2008) work on the Structured Evidential Analysis System, which uses a template-­‐driven approach to mapping evidence, and van Gelder’s (2002) work on Argument Mapping provides disciplined mapping techniques that aid in critical thinking. The influential work of Heuer (1999) on Analysis of Competing Hypotheses (ACH) was designed specifically to prevent analysts’ confirmation bias through matrix analysis of competing hypotheses (ie. potentially plausible “narratives”, in our earlier terminology). ACH-­‐based tools clearly scaffold knowledge-­‐building discourse in a disciplined way, for instance, CACHE (Convertino et al. 2008; Shrager, et al. 2010), provides a collaborative ACH space for the exploration of hypotheses in open science, uses notification spreading through provenance chains in order to simplify revision updating, marking questionable results, and informing scientists when their hypothesis or claims potentially need to be revised. Interestingly, just as intelligence analysis tools such as ACH provide support for detailed analysis of competing options that might be expressed in a collaborative discourse platform such as Cohere, we note that Smallman (2008, p.334) notes the need within the intelligence analysis community for better support in argument analysis and visualization, of the sort provided by Cohere, or Rationale (van Gelder 2002). p. 7

Learning, both personal and collective, is tightly coupled to the concept of CCI, given the emphasis on conceptual change, substantive reasoning, and plausibility. Within CSCL, many theories of learning draw attention to the central importance of different forms of “talk” in the mediation and construction of learning, and the potential role of collaboration tools. p. 7

Andriessen et al. (2003) explore in detail the role of computer-­‐supported argumentation in “confronting cognitions”, p. 7

Mercer (2004) has developed an influential sociocultural theory of learning which draws attention to the central role of dialogue. Validated in the analysis of online discussion, Mercer’s categories of Disputational, Cumulative and Exploratory talk echo many of the forms of discourse we see in both organisational and public online platforms, with Exploratory talk reflecting the most reasoned form of discourse that promotes the open, critical learning orientation that is likely to build effective CCI. p. 7

Having presented the rationale for a CCI infrastructure grounded in reflective discourse, we turn to one strategy for realizing this: the grounding of discourse in source documents through annotation. p. 8

Extending the definition of document to more recent forms of online textual communication within and beyond an organisation, it seems safe to assert that documents will often be the starting points for discussions worth capturing in a CCI platform. p. 8

Levy and Marshall (1995) note the importance of annotation in intellectual work as follows: p. 8

“Annotation is a key means by which analysts record their interpretations of a particular document; in fact, annotation often acts as the mediating process between reading and writing. p. 8

Annotation is a means to make metacognitive activity explicit, to reflect on personal reflection. This activity is particularly important when dealing with information-­‐intensive intellectual tasks, which require powerful scaffolding of thinking and reflection (Lin et al. 1999). By annotating texts, analysts record their reflections and therefore can use them to further reflect on their own thinking and understanding. p. 8

At the same time, within a social environment, annotations can be used to disclose multiple perspectives and to inspire new thoughts, or to enrich the work of others. Recent studies argue that social annotation has a positive effect on several sensemaking tasks such as reflection, self-­‐analysis and examination of changes (Rich and Hannafin 2009), assessment and learning (Kalnikaité and Whittaker 2008). p. 8

social constructivist theories, provide a conceptual framework in which knowledge construction is a highly social activity and it requires “the dialectical interplay of many minds, not just one mind” (Goodman 1986, p.87). Therefore, in order for new knowledge to be constructed, reflections and annotations need to be shared within a community in a social interaction environment. Individual expertise needs to be combined to build distributed expertise, which is constructed through social collaboration and artifacts (Brown et al. 1983; Pea 1993). p. 9

This is well exemplified by Lin et al. (1999) with respect to peer editing conferences that help the authors shape a written product: “The feedback from the group sharpens and guides reflection by the author, leading to revision of what was written in ways beyond what would be possible if the individual was limited to his or her own thoughts. Reflection, therefore, while individual at one level, can also be reflective social discourse.” p. 9

In the discourse-­‐centric annotation framework that we propose, annotation is carried out with two technologies: 1. automatic text analysis, which detects discourse conveying contrasting ideas within documents 2. a platform, which provides users with a structure for annotating documents as well as with tagging and querying functionalities p. 9

Discourse-­‐centric annotations, in this view, come to serve as shareable, improvable artifacts (Scardamalia 2002), and objects for reflection. p. 9

(see Malone et al. 2009 for a useful categorization of different CI technologies) p. 10

Sensemaking, in Russell et al’s view, is the process of identifying a schema which helps to structure and understand information. p. 10

Machine annotation produces two main kinds of output as visual artifacts: sentences and labels. Sentences represent salient contents extracted from the document, and the labels indicate the semantics of the link between the salient content and the document or part of the document. p. 11

Therefore to effectively support sensemaking we aim at enabling analysts to generate a structure which works like a hypothesis about the connections among data, and which helps analysts to construct their narrative. This narrative is visualized so that it can become an artifact for social sharing and further reflection in the analysts’ sensemaking process. p. 11

In our conceptual model stages 2 and 3 produce visual artifacts to be used in stage 4 for making connections: Both machine and human annotation result in two kinds of visual representations: what are termed ideas—which are either extracted from the document (stage 2) or added by analysts (stage 3)—and connections that anchor the ideas in the documents. In stage 4 human and machine annotations are integrated into new visual artifacts which show human and machine annotations together, and critically, which permit the viewing and crafting of meaningful connections between annotations. This semantic p. 11

network can be rendered in many ways, providing materials to inform and provoke wider discourse grounded in documents, but also in the central ideas that have been read into them by the community. p. 12

Ideal sequence of computer and human analysis not fixed, subject to further research. Probably computer-analysis comes first. p. 12

In this latter case, mixed-­‐initiative approaches strongly overlap with the very essence of Collective Intelligence research, which aims at investigating “How can people and computers be connected so that—collectively—they act more intelligently than any individuals, groups, or computers have ever done before?”, 1 p. 12

Compared to Pirolli and Card’s(1995) model of sensemaking, our four-­‐stage model supports the three steps of the foraging loop (External data source, Shoebox and Evidence file) and the first key step of the sensemaking loop, (Schematize). Figure 2 shows how our model (Figure 1) can be mapped into Pirolli and Card’s model of the sensemaking loop. p. 12