The open learner model is the general term for learner (student) models that are accessible to users: visualised and/or interactively maintained. p. 5

the learner model may be considered a collaborative information resource that contains inferences about a student and their learning. Opening the learner model (allowing the student to inspect information about themselves, through visualisation) is a state-of-the art method for supporting students’ learning. According to the SMILI Framework *Bull, 2007b+ there are 13 principal aspects of open learner modelling which apply to the NEXT-TELL project (Section 3): p. 5

Accommodation of the user’s right to view electronic data about themselves Improvement of the learner model’s accuracy Facilitating navigation of the system/navigational materials etc. Increasing learner trust in the system Promoting learner reflection Facilitating (self-) monitoring Supporting planning Encouraging learner control and responsibility in their learning Facilitating formative assessment Provision of a source of information for summative assessment Provision of a source of information for competitive interaction Facilitating collaborative interaction p. 5

Key Stakeholders: (those who have significant influence and importance) Teachers, Students, Other Students (Peers), Parents. Primary Stakeholders: (those ultimately affected) Policy Makers, School Administrators , Researchers, Teachers, Students, Other Students (Peers), Parents.  Secondary Stakeholders: (those indirectly affected) IT Staff, Governing Bodies, University Selection Committees, Future Employers. p. 5

Primary stakeholders who do not have personal contact with the student are interested in monitoring learner models for tertiary purposes (e.g. informing on the efficacy of policy, substantiating claims or validating hypotheses). p. 6

All stakeholders will have partial access to OLMs; teachers will have full access. Key stakeholders and researchers will have access to knowledge related attributes. Peers will able to view the learner model with the permission of the student (and teacher). School admin and policy makers may view general group data. p. 6

All may inspect learner model data. Key stakeholders may add evidence. p. 6

Sources of learner model information include (Section 6.4): The learner model will model different information facets, including: knowledge, difficulties, misconceptions, knowledge level, motivation/affect, reflection/metacognition, self-assessment/epistemological, autonomy, extra-curricular engagement, communication skills, leadership skills, preferences and learning strategies. The learner model may be updated through automated inferences (directly resulting from human-computer interaction) or appraisal information/external assessment tendered by one of the key stakeholders (Section 6.5). Inferences from different sources will be combined; each learner model facet will contain information from many sources. To model effectively, weightings will be placed on information and may be specified by the teacher. Weightings on older information will be depreciated as information is superseded. e-portfolio (video, pictures, documents, weblogs etc.); data artefacts (non-portfolio e.g. Google Docs); self-assessments / appraisal information; (adaptive) learning environment information / other learner models; virtual environments (e.g. OpenSimulator); social platforms (e.g. Facebook); additional evidence tendered directly to the NEXT-TELL system; marked work / external assessment; appraisal information; manual assessment of data artefacts (including e-portfolio etc.). p. 6

We use the term ‘learner model’ rather than ‘student model’ because workpackage 4 focuses on learning both inside and outside the school setting (learners may undertake activities that are not primarily part of their ‘student’ role, but where the outcomes still provide input to their model). p. 8

The initial focus is on knowledge, skills and abilities (KSAs), expanding to epistemic beliefs and finally information about knowledge building, as the project progresses. p. 8

First idea about opening up the learning model appeared p. 9

Self [Self 1988] suggested this as a possibility and, indeed, a right of the user. Open learner models (OLM), although not yet so named, started to appear (e.g. [Bull, 1995b; Corbett, 1995; Kay, 1995]). While designed primarily to in Sections 3.1.1 - 3.1.12. 3.1.1 Accommodate the User’s Right to View Electronic Data about Themselves This point reflects Self’s *Self, 1988+ argument about user rights for accessing their data, raised in the introduction. The OLM approach automatically addresses this issue as it is precisely the data about the user that is externalised. However, there remain issues of how much of the model is made available, and how it is made understandable for the learner. Accommodating the user’s rights needs to be considered alongside the educational aims of supporting learning, which require consideration of pedagogical issues relevant to the particular context of the OLM deployment. For example, if it is educationally more effective to withhold specific learner model information at a particular time, how should this be reconciled with the user's right to see their data when they wish to? In NEXT-TELL, we conceive this principle will differ for data about learners and data that might conceivably be held about parents, with reference to their use of their child's OLM. A learner may access a peer's model and the system will record this event (e.g. to allow teachers to recognise and build upon peer connections in the learning context); information is potentially available to the learner that another learner has viewed their model. 3.1.2 Improve the Accuracy of the Learner Model Improving the accuracy of the learner model relates primarily to cases where the user may provide additional information or corrections to the model, if appropriate. This assumes that the learner model may not always be accurate. There are many reasons that this might be the case, for example: the learner has forgotten; the learner has undertaken reading outside the system; the learner has been guessing (successfully or unsuccessfully); the learner received help during an interaction but is not able to complete a task alone; and so on. Providing a mechanism for users to suggest, negotiate or directly change the model, according to the system goals, maturity of users, etc., can help to improve model accuracy if done appropriately. This, in turn, can improve system predictions of suitable teaching interventions for an individual. Of course, questions remain. For example: can learners assess their own knowledge sufficiently well to provide appropriate (useful) information for their learner model? (Some results suggest that an OLM can help improve self-assessment skills [Mitrovic, 2007]; and some suggest that learners may introduce more accurate changes to their learner model, facilitate learning and metacognitive aspects of learning, these new OLMs were at the same time addressing the issue of the user’s right to access electronic data held about themselves. p. 9

SMILI (Student Models that Invite the Learner In) OLM Framework was developed to help authors and designers describe and analyse their OLMs; compare the features of a range of OLMs; and identify new areas to explore [Bull, 2007b]. p. 9

Accommodating the user’s rights needs to be considered alongside the educational aims of supporting learning, which require consideration of pedagogical issues relevant to the particular context of the OLM deployment. For example, if it is educationally more effective to withhold specific learner model information at a particular time, how should this be reconciled with the user's right to see their data when they wish to? p. 9

Perhaps OLMs could be placed in a framework as a link between adaptive navigation support in educational settings, and intelligent tutoring systems. p. 10

OLMs can be integrated in a learning system, where guidance happens automatically and data is collected, or independent, where learner has to provide data, and the system provides support for choosing learning paths and strengthens ability to self-assess and reflect on own learning. p. 11

OLMs can be used in parallel, or as part of a coordinated approach to formative assessment. ‘Independent OLMs’ are designed in particular to facilitate formative assessment and other metacognitive processes. Given the goals of NEXT-TELL, we expect that the independent OLM will more likely feature strongly as opposed to the more traditional embedded OLM. p. 11

OLMs need not only be accessible to the learner that they represent, but can also be available to peers. This information can be used to prompt competition amongst learners, particularly those who prefer to work alone, but who also like to ensure that their progress is strong compared to the accomplishments of other learners [Bull, 2007c]. p. 12

The UMPTEEN OLM approach allows learners to choose who has access to their data, and whether each of those selected as viewer has named or anonymous access to their learner model. The UMPTEEN OLM has been incorporated into a further independent OLM, OLMlets, where students have chosen to release their own, and view each other’s learner models to support their learning throughout a university module [Bull, 2007a] p. 12

learner models that are accessible to peers can prompt collaborative learning. p. 12

the “Learner Modelling for Reflection” website: http://www.eee.bham.ac.uk/bull/lemore/examples.html. p. 12

OLMs that allow interaction with underlying model p. 13

Flexi-OLM [Mabbott, 2006] and a recent extension to OLMlets [Ahmad, 2010]. p. 13

Inspectable learner models [Corbett, 1995; Mitrovic, 2007]; The user and system maintaining different model attributes [Beck, 1997]; The user adding evidence for the system to take into consideration [Kay, 1995]; The user and system jointly negotiating the contents of the model where separate belief measures are maintained if there is no agreement between the two parties, with the system taking both sets of data into account in adaptation to the learner [Bull, 1995a; Dimitrova, 2003; Kerly, 2008]; The user trying to persuade the system to change the model, providing evidence, for example through testing [Bull, 2007c; Mabbott, 2006]; The system trying to persuade the user, but with the user remaining in control [Grigoriadou, 2003]; Editable learner models [Ahmad, 2010; Mabbott, 2006]. p. 14

To meet the concerns of NEXT-TELL, the OLM will be a kind of hybrid approach: it will be embedded in a larger environment, but one that serves the various stakeholders in a flexible manner. It will also be independent of the typical tutorial or guidance st rategies present in an intelligent tutoring system, as the primary aim with reference to teaching, is for the OLM to support teachers in their decision-making and interactions with students. p. 14

(Collaborative Representations, ConcertChat) p. 21

Meta-design [Fischer, 2003; Fischer, 2010] represents one possible approach to developing the communication and negotiation layer, as outlined in D2.1. Meta-design focuses on design for designers, with the ideal state being the recruitment of some presumably small segment of users into the role of designers. p. 23

We have proposed a triadic model of teaching analytics based on the fusion of PD and DBR [Vatrapu, 2011]. p. 23

Integrating emerging developments in visual analytics and the established methodological approach of design-based research (DBR) in the learning sciences, we introduce a new method called “Teaching Analytics” and explore a triadic model of teaching analytics (TMTA). TMTA adapts and extends the Pair Analytics method in visual analytics, which in turn was inspired by the pair programming model of the extreme programming paradigm. Our preliminary vision of TMTA consists of a collocated collaborative triad of a Teaching Expert (TE), a Visual Analytics Expert (VAE), and a Design-Based Research Expert (DBRE) analysing, interpreting and acting upon real-time data being generated by students’ learning activities by using a range of visual analytics tools. p. 24

As NEXT-TELL does not aim to build an adaptive teaching system, the user’s awareness of the effect of the learner model on personalisation is not relevant in the traditional sense. p. 27

Inferences resulting from students’ interaction contribute to formative and summative assessment, and comprise both qualitative and quantitative components. Supplementary evidence, appraisals and evidence of 21st century skills also form part of the model. Each of these types of information may not necessarily be combined; each may potentially contribute to a different facet of the model. p. 30

Some information sources (e.g. adaptive learning environments) can capture more diverse and fine grained information, which may be modelled and further inferences based upon. For example, information such as learners’ confidence, time on task or frequency of access of reference materials could help infer the affective states of boredom or confusion. These learning states are important, for instance, in determining the zone of proximal development in Vygotskyian pedagogical theory [Vygotsky, 1962]. Other information sources such as marked work may be less rich in information, perhaps only able to contribute to knowledge level information. p. 31

The weightings are likely to vary depending upon the activity and so may need to be specified by the teacher or in a default template that can be edited. Further weightings for learner model information may need to be applied across time. For the learner model to be an accurate model of current understanding the weightings of older evidence should be depreciated. p. 33

Information from different sources will need to be combined. The modelling process will need to deal with the different levels of granularity of the incoming inferences and their contribution to different facets of the learner model p. 33

Ahmad, N., Britland, M., Bull, S. & Mabbott, A. (2010). A Role for Open Learner Models in Formative Assessment: Support from Studies with Editable Learner Models, Proceedings of Workshop on Technology-Enhanced Formative Assessment, EC-TEL 2010. Ahmad, N. & Bull, S. (2009). Learner Trust in Learner Model Externalisations, in V. Dimitrova, R. Mizoguchi, B. du Boulay & A. Graesser (eds), Artificial Intelligence in Education 2009, IOS Press, Amsterdam, 617-619. Balka, E. (2010). Broadening Discussion About Participatory Design: A reply to Kyng, Scandinavian Journal of Information Systems, 22(1), Article 7. Available at: http://aisel.aisnet.org/sjis/vol22/iss1/7 Balka, E., (2005). Tidying up loose ends: Theoretical and practical issues related to women’s participation in technological design. In H. Rohracher (Ed.), User involvement in technological innovation (pp. 147-173), Profil-Verlag, Munich. Beck, J., Stern, M., & Woolf, B.P. (1997). Cooperative Student Models. In B. du Boulay & R. Mizoguchi (Eds.) Artificial Intelligence in Education: Knowledge and Media in Learning Systems, Amsterdam: IOS Press, 127-134. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178. Bull, S. & Kay, J. (2010). Open Learner Models, in R. Nkambou, J. Bordeau & R. Miziguchi (eds), Advances in Intelligent Tutoring Systems, Springer, 318-338. Bull, S., Dong, X, Britland, M. & Guo, Y. (2008). Can Students Edit their Learner Model Appropriately?, in B.P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (eds), Intelligent Tutoring Systems: 9th International Conference, Springer-Verlag, Berlin Heidelberg, 674-676. Bull, S., Mabbott, A., Gardner, P., Jackson, T., Lancaster, M., Quigley, S. & Childs, P.A. (2008). Supporting Interaction Preferences and Recognition of Misconceptions with Independent Open Learner Models, in W. Neijdl, J. Kay, P. Pu & E. Herder (eds), Adaptive Hypermedia and Adaptive Web-Based Systems, Springer-Verlag, Berlin Heidelberg, 62-72. Bull, S. & Britland, M. (2007). Group Interaction Prompted by a Simple Assessed Open Learner Model that can be Optionally Released to Peers, in P. Brusilovsky, K. Papanikolaou & M. Grigoriadou (eds), Proceedings of Workshop on Personalisation in E-Learning Environments at Individual and Group Level (PING), User Modeling 2007. Bull, S. & Kay, J. (2007). Student Models that Invite the Learner In: The SMILI Open Learner Modelling Framework, International Journal of Artificial Intelligence in Education 17(2), 89- 97. Bull, S., Mabbott, A. & Abu-Issa, A. (2007). UMPTEEN: Named and Anonymous Learner Model Access for Instructors and Peers, International Journal of Artificial Intelligence in Education, 17(3), 227-253. Bull, S., Quigley, S. & Mabbott, A. (2006). Computer-Based Formative Assessment to Promote Reflection and Learner Autonomy, Engineering Education: Journal of the Higher Education Academy Engineering Subject Centre 1(1), 8-18. Bull, S., Mangat, M., Mabbott, A., Abu Issa, A.S. & Marsh, J. (2005). Reactions to Inspectable Learner Models: Seven Year Olds to University Students, Proceedings of Workshop on Learner Modelling for Reflection, International Conference on Artificial Intelligence in Education 2005, 1-10. Bull, S., & Pain, H. (1995). Did I say what I think I said, and do you agree with me? Inspecting and Questioning the Student Model. In J. Greer (ed.) Proceedings of the World Conference on Artificial Intelligence in Education. Charlottesville, VA, 501-508. p. 42

Bull, S., Pain, H. & Brna, P. (1995). Mr. Collins: A Collaboratively Constructed, Inspectable Student Model for Intelligent Computer Assisted Language Learning, Instructional Science 23(1-3), 65-87. Collins, A. (1992). Toward a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). Berlin, Germany: Springer-Verlag. Corbett, A.T., & Anderson, J. (1995). Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge, User Modeling and User-Adapted Interaction 4, 253-278. De Bra, P., Brusilovsky, P., Houben, G.-J. (1999). Adaptive Hypermedia: From Systems to Framework. ACM Computing Surveys 31(4) Dimitrova, V. (2003). StyLE-OLM: Interactive Open Learner Modelling. International Journal of Artificial Intelligence in Education, 13(1), 35-78. Fischer, G. & Hermann, T. (2010): “Socio-Technical Systems - A Meta-Design Perspective”, International Journal for Sociotechnology and Knowledge Development (in press) Fischer, G. (2003). Meta-Design: Beyond User-Centered and Participatory Design, Proceedings of HCI International 2003, J. Jacko and C. Stephanidis (Eds.), Crete, Greece, June 2003, pp. 88-92. Grigoriadou, M., Tsaganou, G., & Cavoura, T. (2003). Dialogue-Based Reflective System for Historical Text Comprehension. In Proceedings of the Workshop on Learner Modelling for Reflection (Supplemental Proceedings Vol 5, pp. 238-247). International Conference on Artificial Intelligence in Education, Sydney, Australia. Jungk, R., & Müllert, N. (1987). Future workshops: How to create desirable futures. London, United Kingdom: Institute for Social Inventions. Kay, J. (1997). Learner Know Thyself: Student Models to Give Learner Control and Responsibility, International Conference on Computers in Education, Charlottesville, VA: AACE, 17-24. Kay, J. (1995). The um Toolkit for Cooperative User Modeling. User Modeling and User- Adapted Interaction, 4(3), 149-196. Kensing, F., & Madsen, K. H. (1991). Generating visions: Future workshops and metaphorical design. In J. Greenbaum & M. Kyng (Eds.), Design at work: cooperative design of computer systems (pp. 155-168): CRC Press. Kensing, F. (1987). Generation of visions in systems development: a supplement to the tool box. Paper presented at the The IFIP TC 9/WG 9.1 Working Conference on system design for human development and productivity: participation and beyond on System design for human development and productivity: participation and beyond, Berlin, Germany. Kerly, A., Ellis, R. & Bull, S. (2008). CALMsystem: A Conversational Agent for Learner Modelling, Knowledge-Based Systems 21(3), 238-246. Kerly, A. & Bull, S. (2007). Open Learner Models: Opinions of School Education Professionals, in K. Koedinger, R. Luckin & J. Greer (eds), Artificial Intelligence in Education, IOS Press, Amsterdam. Kyng, M. (2010). “Bridging the Gap Between Politics and Techniques: On the next practices of participatory design,” Scandinavian Journal of Information Systems, 22(1), Article 5. Available at: http://aisel.aisnet.org/sjis/vol22/iss1/5 Mabbott, A. & Bull, S. (2006). Student Preferences for Editing, Persuading and Negotiating the Open Learner Model, in M. Ikeda, K. Ashley & T-W. Chan (eds), Intelligent Tutoring Systems: 8th International Conference, Springer-Verlag, Berlin Heidelberg, 481-490. Mitrovic, A., Martin, B. (2007). Evaluating the Effect of Open Student Models on Self- Assessment, International Journal of Artificial Intelligence in Education 17(2), 121-144. p. 43

Pain, H., Bull, S. & Brna, P. (1996). A Student Model 'For Its Own Sake', in P. Brna, A. Paiva & J. Self (eds), Proceedings of the European Conference on Artificial Intelligence in Education, Lisbon, 191-198. p. 44

Perez-Marin, D. & Pascual-Nieto, I. (2010). Showing Automatically Generated Students’ Conceptual Models to Students and Teachers, International Journal of Artificial Intelligence in Education, 20(1), 47-72. [Schmidt-Belz, 2005] Schmidt-Belz, B. (2005). User Trust in Adaptive Systems, Proceedings: Lernen, Wissensentdeckung und Adaptivität (LWA), http://www2.dfki.de/lwa2005. [Self, 1988] [Vatrapu, 2011] [Vygotsky, 1962] [Weber, 2001] Self, J.A. (1988). Bypassing the Intractable Problem of Student Modelling, Proceedings of Intelligent Tutoring Systems, Montreal, 18-24. Vatrapu, R., Teplovs, C., Fujita, N. & Bull, S. (2011). Towards Visual Analytics for Teachers’ Dynamic Diagnostic Pedagogical Decision-Making. Paper to appear in the Proceedings of the 1st International Conference on Learning Analytics and Knowledge, Banff, Alberta, Canada. Vygotsky, L. (1962). Thought and language. Cambridge [Mass] : MIT Press. Weber, G., Brusilovsky, P. (2001). ELM-ART: An Adaptive Versatile System for Web-based Instruction, IJAIED 12, 351–384. p. 44