ECAAD is a method and a set of tools that teachers will be able to use to conduct new forms of formative assessment and integrate these into their teaching plans. p. 5

In formative assessment a learner’s activities are frequently evaluated with the aim of adjusting or improving instruction (teacher) and learning (student). Hence, formative assessment requires an authentic context, data collected from multiple sources (such as portfolios, self-appraisals, presentations, etc), and global and specific diagnosis aimed at providing helpful feedback. p. 5

ECAAD stands for Evidence-centred Activity and Assessment Design p. 5

ECAAD provides three elements for the other NEXT-TELL layers: (a) it provides a ‘script’ for the learning activities and the (mostly embedded) assessment processes that are to be realized in the Learning Environment, (b) it provides information on what aspects of students’ learning activities need to be tracked and which aspects of students’ work need to be stored in the e-portfolio, and © it provides information on how to transform data on students’ activities and products into information about students’ competencies that can be rendered in the Open Learner Model. p. 5

three areas of research which are stepping stones for the project: the Cognitive Density concept, Evidence-centred Assessment Design methodology, and research on Learning Design p. 5

Cognitive Density is the parameter we suggest as the critical factor to decide if a teacher together with the NEXT-TELL tools is successful; in short, it is a measure of engagement with subject matter, learning activities, and peers. p. 5

he ECAAD Learning Activity Planner and the ECAAD Assessment Method Designer. p. 5

For STEM, we develop in particular the case of Understanding Visual Representations (of data and of information), and for TESL we develop the case of English Conversation skills. p. 5

We have conducted a domain analysis of the underlying skills, and performed a first modelling of these skills in terms of Knowledge Space Theory. The current analysis yielded about 100 skill components. p. 5

For TESL, two scenarios have been developed for the case of writing and for spoken conversation. In the writing scenario, students will write (individually and/or collaboratively) in Google Docs, developing an essay over multiple stages of revisions (optionally involving peer review). In the conversation-training scenario, students will be engaged in role plays in an avatar-populated immersive environment (such as Second Life). For both cases, we identified two approaches to formative assessment: The first one builds on the Common European Framework of Reference for Languages (CEFR), a comprehensive set of assessment criteria for written and spoken communication. Teachers will be supported in applying this assessments method to writing assignments by semi-automatic application of the CEFR framework to essays, and in the conversation scenario by supporting the tagging of ‘video’ recordings of students’ performance in the immersive world. The second one builds on Epistemic Network Analysis, a 21st Century assessment method that provides the teacher with a p. 5

holistic view on students’ learning. Students’ essays and their conversations in the immersive environment, respectively, will be increasingly automatically analysed by providing teachers and students with an Epistemic Network Analysis tool. p. 6

The assessment map represents the high level view on assessment methods developed in a semi-formal/abstract representation. The representation of assessment methods in an assessment map provides an overview on the structure of a method and its purpose p. 9

As a break-down/drill-down of an assessment method in the assessment map, the assessment model provides details on the method, the necessary data sources and calculation schema applied in the form of a workflow orchestrating on a technical level web-services that provide data, calculate and present information. p. 9

ECAAD is conceptually grounded in Evidence-centered Assessment Design (ECD) (Mislevy 1999; Mislevy & Haertel, 2006; the PADI project) p. 9

The ECAAD Assessment Method Designer provides IT support to establish the assessment map as well as the detailed break down by using graphical modelling as an externalisation approach. p. 9

The ECAAD Activity Sequence Planner provides IT support in planning learning activities and sequences. Similar as for assessment method design a top-down approach is proposed, starting conceptually with the learning domain map and breaking this map down to an overview map (Learning Sequence Map) and further on to learning Sequence Models. p. 9

The Learning Domain Map provides a goal driven overview on the learning scenario and the defined paths to reach a certain objective. The main objective of the map is to allow an environmental analysis and dependencies in the specific scenario. p. 9

The Learning Sequence Map enables high-level/abstract modelling of learning sequences. This map provides the abstract representation for a later breakdown p. 9

The Open Models initiative (www.openmodels.at) targets the development of modelling methods using ICT support by applying metamodelling concepts. Open Models in NEXT-TELL is regarded as the collaborative engineering platform for all modelling aspects in the project. p. 10

Visual and interactive representation of a learner’s model p. 10

TISL p. 10

Teacher Inquiry into Students’ Learning p. 10

the key question that NEXT-TELL wants to answer: How can ICT be put to use by teachers to support (and innovate) formative assessment and planning for learning in the ICT-rich classroom? p. 11

Formative assessment is a vital element of the pedagogical repertoire for preparing 21st Century students. (Shute, 2008) writes that in formative assessment the results of a learner’s activities are frequently evaluated with the aim of adjusting or improving instruction (teacher) and learning (student). Hence, formative assessment requires an authentic context, data collected from multiple sources (such as portfolios, self- appraisals, presentations, etc.), and global and specific diagnosis aimed at providing helpful feedback. (Cowie Bell 1999) define formative assessment as a bidirectional process between teacher and student to enhance, recognize, and respond to the learning. p. 11

Formative assessment is considered a promising approach to enable 21st century teaching since it potentially promotes self-reflection and self-directed learning processes and, more importantly, it facilitates the integration of new subject-specific knowledge into the student’s existing knowledge network. It also helps the teacher adapt the educational processes to the individual needs and, therefore, making formal education more effective and also more enjoyable (perhaps “less boring”). p. 11

Building on established models of formative assessment (such as Popham, 2008), and of assessment in general (Mislevey et al., 1999), NEXT-TELL will support the teacher in the technology-rich classroom. p. 11

As a construct to measure how successful the system teacher-plus-technology is in supporting classroom learning, we have identified cognitive density. p. 11

The notion of cognitive density is, hence, central to our framework as it captures an emerging quality of a system of interacting human and technical actors engaged in the bringing about of learning. p. 12

"in the classroom" - how does this transfer to online spaces? p. 12

Cognitive density describes ”… the aggregate level of students' engagement with learning materials and thinking, their progress in learning, their communication and their use of time – that is, productive activity in the classroom at any given time.” (Crawford et al., 2008, p. 1). Cognitive density, thus conceived, is a general attribute of a learning environment and hence not specific to any specific pedagogical intervention: every classroom pedagogy will create cognitive density, but not necessarily on the optimal level. p. 12

Cognitive density itself cannot be directly controlled, but is a second-order effect of more controllable and measurable constructs: communicative, content, and temporal density of the learning environment (see Figure 2). The goal of classroom technology is not to maximize these densities at the different layers (e.g., the number of events per unit of time), but to optimize them for particular curricular or pedagogical objectives, taking into account information from individual students in a dynamically changing learning environment. p. 12

content density -> temporal density -> communicative density -> cognitive density p. 12

The fact that CD cannot be directly controlled is due to the fact that it is (a) not solely a individual process and (b) to the extent that it is a individual cognitive process neither teachers nor students have direct control over it. Strictly speaking, nobody can decide to “learn” (or not to learn, for that matter)–we can only decide to engage with content in a manner that leads with some likelihood to what is considered accountable learning outcomes. p. 13

The notion of emergence also indicates that CD cannot be seen as a “dependent variable”, as being caused by content, temporal and communication density. This because CD is connected via feedback loops to these processes: what is an optimal level of content density, for instance, will depend amongst other things on how students engage cognitively with the content. The relation between all these 'variables' is systemic rather than linear. p. 13

The layered model also points to the fact that “learning improvements” as for instance measured by achievement tests is even less under control of the teacher than CD. While teachers can create conditions approximating optimal levels of CD, there is no simple linear relation between those and achievement outcomes. Instead, the relation is mediated by numerous 'factors' such as the appropriateness of the test to assess the kind of learning that took place in students, and many situational factors such as test anxiety or tiredness. p. 13

The emergent, distributed nature of CD makes it challenging to measure it in any straightforward sense. Any measurement will require to combine measurements of individual proximal indicators (subjectively felt engagement and 'flow', motivation to continue, objective measures such as activities/unit of time) with indicators that describe the social interactions in the classroom and their distribution. At this stage, not measurement framework for CD exists, and it will be part of the research in NEXT-TELL to develop one. One promising avenue seems to build on work in motivational psychology to measure peoples' flow experience. p. 13

Interesting concept in relation to asynchronous collaboration and synchronous meetings p. 13

Communicative density refers mainly to the bandwidth through which instructional messages can be exchanged. The well-known bandwidth limitations of the face-to-face classroom (where only one person can speak at any point in time) is extended dramatically as 1:1 computing enters the classroom. But obviously using that bandwidth to the fullest extent is not pedagogically advisable. Instead, the communication density will need to be carefully managed, dependent on task and learner demands. p. 13

Again, simple heuristics, such as to provide feedback as fast as possible, do not lead to optimal cognitive density. In many situations, it might be better to delay feedback. p. 13

Principled Designs for Inquiry (PADI, padi.sri.com) p. 14

ECD builds on the idea that assessment means to establish claims about students’ knowledge and skill grounded in evidence (behaviour in assessment tasks) and supported by warrants, see Figure 3. ECD is grounded in a view of assessment as an argumentative process, comparable to the (Toulmin, 1958) scheme of argumentation. Evidence takes the form of systematic observations on students’ performance (e.g., when playing an instrument) and work products (e.g., problem solutions, essays, sketches, media artefacts), while warrants provide justifications for drawing inferences (claims) from such observations. A warrant can take for example be provided in form of a research study that has established a relation between particular performance and particular skills (the latent construct; assessment claims are typically about latent constructs: skills, knowledge, aptitudes, motivational dispositions, learning styles, etc.). p. 14

Interesting to think about assessment as argumentation-driven if the class itself is using argumentation as a way of learning. Any link? p. 14

the general processes of getting from observations on students (test) performance to claims about their knowledge, skills, and abilities (KSAs) requires from the assessment designer to specify (a) a task model (what the student will do), (b) an evidence model (what aspects of the student performance are relevant for assessment), and © a student model (how the evidence is to be transformed and combined to yield a value for a KSA element). p. 14

The ECAAD method to be developed in NEXT-TELL generalizes the ECD method in a number of ways. The most important generalization is to not limit the method to summative assessment; formative assessment, in order to be rigorous, also needs to be based on an explicated method. p. 16

A further generalization concerns the content and form of the student model: it should be open and allow to represent structural qualitative information about a student as well as allow to represent aspects of students besides knowledge and skill: values, epistemic orientations, emotional relations to subject matter for instance. Thirdly, the evidence based needs to be widened: ECAAD will incorporate evidence not only from assessment tasks (as ECD/PADI does), but relate to learning and work performance and products in a more general sense, thus allowing the inclusion of evidence from e-portfolios (product focus) and log-files that captures students’ use of learning software/software used for learning (process focus). Fourthly, the generalized method will be applied to learning in areas other than science, namely language (English as a first/as a foreign language) learning. Finally, in NEXT-TELL the ECAAD method will be used as a basis for communication and negotiation with stakeholders and as a basis for teachers’ professional learning—learning about students’ learning. p. 16

learning design (LD). LDs are “… generic approaches to the structuring and orchestration of learning activities” (Falconer, Beetham, Oliver, Lockyer, & Littlejohn, 2007). Learning designs have been researched from a computer science perspective in recent years (e.g. Koper & Tattersall, 2005) because of the interest in re-usable instructional components to deliver on-line learning, e-learning in particular. Learning designs from this perspective have also been called practice models: “Common, but decontextualised, learning designs that are represented in a way that is usable by practitioners (teachers, managers, etc.)” (Falconer et al., 2007, p. 2). p. 16

Research shows that teachers indeed find formal designs of instruction uninteresting. Which leads (Falconer et al., 2007) to the formulation that there is often a gap between the inspirational function of a design and its implementation function (runnable versus inspirational representations). p. 16

The amount and structure of information required to reuse a learning design is too complex for a single representation. Multiple representations that convey information in different ways are necessary. p. 16

A number of notations (or more general representations, including video) have been suggested and been analyzed for learning designs, amongst them: Case studies and video case studies Controlled Vocabularies Matrices/templates Patterns Concept maps Temporal sequences Flow diagrams (Falconer et al., 2007) discusses and compares these in some detail, so that we can confine ourselves to an overview here. p. 17

Controlled vocabularies (taxonomies, ontologies) for instructional planning p. 18

(Currier, Campbell, & Beetham, 2005) p. 18

DialogPLUS for learning activities (Conole & Fill, 2005) p. 18

Concept maps that use a controlled vocabulary can be considered to be semantic nets, and as such can be interpreted by programs (Paquette, 2003) p. 19

constructing semantic nets amounts to a knowledge engineering task, not something teachers are trained in or have the time for. More ‘informal’ concept maps are frequently used, however, due to their effectiveness for learning (Novak & Gowin, 1984), p. 19

ECAAD uses the Open Models4 approach (Karagiannis et al., 2008) as the methodological basis for modelling assessment processes and learning activities. The argument for modelling in TEL has been made extensively before, and does not need to be repeated here in detail (e.g., Koper, 2005). p. 20

the rise of various (mostly graphical) languages for learning design (Botturi & Stubs, 2007) p. 20

models of teaching, models of assessment, connecting the two p. 20

general use case p. 20

where a teacher6 analysis learning requirements by performing a domain analysis, plans a sequence of teaching (what the teachers does) and learning (what the students do) activities, and integrates assessment7 methods into the sequence of learning activities p. 20

Open Models are different from Open Learner Models (see D4.1). While the later refer to (usually graphical) computer-presented representations of a learners proficiencies, the former refers to a general approach to model processes; they could be called Open Process Models, but the more general term Open Models is used in the research community. These two uses of the word ‘model’ are rooted in different research communities: those for adaptive learning systems (learners models) and those of business IT (process models). p. 20

In principle (though rarely in practice) students can take on the role of a teacher in the sense that they plan for their own learning. There is nothing in ECAAD that would not allow usage of methodology and tools by students. p. 20

ECAAD methodology does not specify: it neither specifies certain pedagogy (a strategy for teaching) nor does it specific learning activities (the tactical level). Also, it does not, in principle, specify specific assessment methods. p. 21

within NEXT-TELL we will develop exemplary scenarios for the use of our methods and tools p. 21

ECAAD modelling method p. 21

ECD sees assessment (any form of assessment) as the process of getting from questions about students' knowledge and learning to activities in learning environments (top-down view), and of how then observations on students' learning can be propagated upwards through the evidence model to a model of a student's proficiencies. p. 23

During domain analysis (conducted by teachers), learning goals (which would typically come from a curriculum plan) describing student competencies are analysed and set in relation to prerequisite knowledge, skills, and abilities (KSAs). A domain analysis, thus, yields firstly an identification of the KSAs that go into the target competencies (learning goals), and secondly leads to the identification of pre-requisite relationships amongst these KSAs. Such a structure can be conveniently represented as a directed graph, a KSA map. Thirdly, and as extension to the typical KSA (pre-requisite relations) map, the teacher should be supported in specifying not only the correct knowledge, but also possible misconceptions (for declarative knowledge) and/or “bugs” in procedural knowledge; in other words, there needs to be room for specifying what can go wrong with learning, based on knowledge about what does frequently go wrong. Fourthly, the domain map should contain information about learning trajectories: likely paths for learning to progress over time, including likely ‘detours’ due to misconceptions and/or buggy procedural knowledge. p. 24

The Learning Activity Map is a high-level description of the teaching/learning sequence students (or a specific student) will undergo to reach the learning goal specified in the Domain Map. The LAM contains information on the general pedagogy (e.g. problem solving, inquiry, knowledge building) with rationale, as well as an outline of the teaching/learning activities, resources involved, and ICTs involved. Furthermore, it does include trigger points for assessment, and a specification of the kind of assessment method required, again with rationales for these decisions. These design rationales correspond to warrants in an argumentative logic; they provide justifications for why students should engage in a specific learning activity, and not in others. The learning activity map can also specify alternative learning activities, i.e. activities which would serve the pedagogical purpose as well or nearly as well. p. 25

A NEXT-TELL assessment map is a semi-structured description of how to assess a specific knowledge/skill/ability (KSA), including 21st Century KSAs, using learner performance/product data that come from ICTs. An assessment map is semi-structured in as far as key attributes are pre-specified p. 28

an Assessment Model needs to specify the following elements: p. 30

Collect p. 30

Define the data adaptors to be used, including adaptors to those data that can from human raters (teachers, peers, others) p. 30

Filter  When data come from logfiles etc., filter for the data needed for the KSA diagnosis p. 30

 Transform  Raw data may need to be transformed, e.g. normalized; observational data may need to be mapped into categories, using check lists, rubrics etc. p. 30

 Diagnose  Data in the widest sense are interpreted in terms of assessment categories, that is, in terms of KSAs. There are multiple ways to do this, e.g. in terms of mastery of a KSA, of social comparison (rank order), with respect to a population (rank order, percentile), in comparison with expert solutions, etc. p. 30

 (Display)  How to display/render a diagnosis in NEXT-TELL in principle to be decided in the Open Learner Model. However, in many cases a basic representation will be delivered by the assessment method as a by-product of the Diagnose and/or Combine step p. 30

In the final step (7), the hand-over to the execution environment, i.e. the students’ learning environment needs to be achieved. This delivery is made available through specific export interface that translate the semi-formal models into an execution language. Candidates for such execution languages are SCORM, DITA, IMS LD, and QTI. The main objective is to provide an automatic export and execution feature, meaning the abstract representation of the model is translated into the execution graph on the fly, by using predefined execution chunks, identified during design time.   Req. No.  Formulation  Rationale   INTEGRATION -MA1  Cross domain dependency view on activity sequence and assessment method, impact view on KSA Analysis and Simulation  INTEGRATION -MA2  Validation of models with regards to cardinalities Workflow control flow p. 32

The approach we want to utilize for NEXT-TELL – in form of software services – is based on the probabilistic, non-numerical method of Competence-based Knowledge Space Theory (CbKST). CbKST originates from KST established by (Doignon & Falmagne 1985, 1999), which is a well-elaborated set-theoretic framework for addressing the relations among problems (e.g., test items). It provides a basis for structuring a domain of knowledge and for representing the knowledge based on prerequisite relations (see Figure 14 for an example). While KST focuses only on performance (the behaviour; for example, solving a test item), CbKST introduces a separation of observable performance and latent, unobservable competences, which determine the performance. p. 34

Albert, Kickmeier-Rust & Matsuda, 2008 p. 34

An empirically well-validated approach to CbKST was introduced by Klaus Korossy (1997, 1999); basically, the idea of the Competence-Performance Approach (CPA) is to assume a finite set of more or less atomic competences C = ,a, b, c, d, …- (in the sense of some well-defined, small scale descriptions of some sort of aptitude, ability, knowledge, or skill) and a prerequisite relation between those competences. A prerequisite relation states that competence a (e.g., to multiply two positive integers) is a prerequisite to acquire another competence b (e.g., to divide two positive integers). If a person possesses competence b, we can assume that the person also possess competence a. To account for the fact that more than one set of competences can be a prerequisite for another competence (e.g., competence a or competence b are a prerequisite for acquiring p. 34

competence c), prerequisite functions have been introduced, relying on and/or-type relations. A person’s competence state is described by a subset of competences of C, for example {a, b}. Due to the prerequisite relations between the competences, not all subsets of competences are possible competence states. To give an example, imagine four competences from the domain of basic algebra, the abilities to add, subtract, multiply, and divide numbers. Given four competences, the set of all possible knowledge states is 24. If we assume that the competences to add, subtract, and multiply numbers are prerequisites for the competence to divide numbers, not all of the 16 competence states are plausible. For example, it is highly unlikely that a child has the competence to divide numbers but not to add numbers. The collection of possible competence states corresponding to a prerequisite relation is called competence structure. Such competence structure also singles out different learning paths for moving from the naïve state { } (having no competences of a domain) to the state of possessing all of a domain’s competences C. p. 35

So far, the structural model focuses on latent, unobservable competencies. By utilizing interpretation and representation functions the latent competences are mapped to a set of tasks (or test items) Q = ,p, q, r, s, …- relevant for a given domain. Through the aforementioned example, Q might include a number of addition, subtraction, multiplication, and division tasks. The interpretation function assigns a set of competences required to solve a task to each of the problems in Q. Vice versa, by utilizing a representation function, a set of problems is assigned to each competence state, which can be mastered in this state. This assignment induces a performance structure, which is the collection of all possible performance states. Due to these functions, latent competences and observable performance can be separated and no one-to-one mapping is required. Moreover, learning or development is not seen as a linear course; equal for all children, rather, development follows one of a set of individual learning or developmental paths. More in-depth introductions to CbKST can be found, for example, in Doignon & Falmagne (1999). p. 35

In the context of NEXT-TELL, this approach to the modelling of the knowledge domain in the first instance and of meaningful learning sequencing in a second instance provides some key advantages: One the on hand, we have a formal and computable method to analyse and define domains (i.e., the subject matter) on the level of (more or less) atomic entities of knowledge or ability – what term competency here. This formal nature in combination with a set of available tools and procedures enables the teacher to develop an individual structure of the subject matter, an individual curriculum, that involves the information and task from multiple sources (e.g., textbooks, books, websites, multimedia sources, peer activities, etc.). p. 35

The knots of this Hasse diagram indicate meaningful competence states of a student while the edges indicate admissible transitions from one competence state to another by acquiring another competency. p. 35

The Hasse diagram and associated graphs may be used to visualize the knowledge of students in a non- numerical and precise manner. This opens (a) a link to the use of open learner models (OLM) and it may serve (b) the ideas of negotiating and communicating appraisal outcomes (towards students and parents). The notions of inner and outer fringes enable very clear statements of what a student knows/can do at this very moment and it defines very clear indications of what a student can/should learn or practise next. The CbKST approach originates from the field of intelligent and adaptive tutorial systems, that is, autonomous computer algorithms that attempt to interpret learner needs, learning progress, and problems within learning episodes and tailor educational measures accordingly. The aim is, essentially, to provide individual students with personalized and psycho-pedagogically optimal learning experiences and environments. The challenge within NEXT-TELL is, to translate this “computer-centered” approach to the 21st century classrooms and the use of human teachers. In addition, the approach has to be mapped and linked to the other important aspects and facets of NEXT-TELL such as OLM, ECAAD, or TISL. p. 35

Translating ideas from intelligent tutor systems to classrooms and collaboration - but underlying structure of knowledge still fits - what about more fuzzy knowledge/skills? p. 35

Domain analysis. Taking into consideration on the one hand the education situation in the different countries and on the other hand the theoretical base of graph comprehension the domain “understanding of and working with visual representations” was decomposed into a catalogue of skills or competencies. This process of domain analysis displayed a set of in total 101 skills which specify and circumscribe as detailed as possible the envisaged domain of knowledge. p. 38

The skill definition is characterized by a pair consisting of a ‘concept’ and an ‘action verb’. The action component is associated with the categories of the Bloom’s revised taxonomy (Anderson & Krathwohl, 2001). p. 38

CbKST-driven domain structuring. A number of in total 101 skills result in 2101 combinations of skills (for example, 2.535.301.200.456.458.802.993.406.410.752 combinations). To reduce this number of possible competence states significantly, the skills were analysed regarding prerequisite relations among them. Such relation states that a given skill a is the prerequisite for another superior skill b. So a global structure can be derived p. 38

Breaking down 21st century key skills such as creativity or reflection in connection with TESL, means encouraging pupils’ learning and thinking skills by having them communicate in English in a lot of different ways, while supporting them with meaningful examples and feedback. Getting used to working with and towards short-term and long-term goals, should enable the pupils, little by little, to be able to set their own learning goals and plan how to reach these. N p. 40

In order to provide examples for assessment models applicable to language learning (written and spoken English) in the ECAAD Assessment Model Library, the project will develop a criteria-based approach (using the Common European Framework of Reference for Languages) that can be used for both written and spoken communication, and an approach to formative assessment that builds on epistemic network analysis, heralded as a truly 21st Century approach to learning and assessment. p. 40

How context specific do assessment schemes have to be? History and sociology are very different, but are there common measures that show "engagement, depth of thinking etc?" - are there ways of assessing success of a meeting, regardless of the topic of the meeting? p. 41

Epistemic Network Analysis p. 41

Shaffer’s Epistemic Frame Theory (EFT) was developed in the context of developing epistemic games (Shaffer, 2006). Epistemic games as conceptualized by Shaffer build on the Community of Practice concept, using a game format (role plays) to engage students in extended learning (over weeks and months). EFT expresses a theory of learning that “…looks not at isolated skills and knowledge, but at the way skills and knowledge are systematically linked to one another—and to the values, identity, and ways of making decisions and justifying actions of some community of practice” (Shaffer et al., 2009, p. 4). EFT stipulates that any community of practice has a culture that can be expressed as an epistemic frame, a structure composed of skills, knowledge, identity (“the way that members of the community see themselves”), values, and epistemology (“the warrants that justify actions or claims as legitimate within the community”). In short, EFT is proposed as an account for the form and development of professional perspectives and practices, the specific ways of seeing and doing things that come with participation in a community of practice. p. 41

link between epistemology and Toulminen warrants... p. 41

We build on EFT not only because it brings together notions of cognitive learning theories and socio-cultural learning theories, but also because of the method Shaffer has conceived of to describe and assess the development of epistemic frames: Epistemic Network Analysis (Shaffer et al., 2009). It takes in particular into account that learning means the coordinated development of the various frame elements (SKIVE: skills, knowledge, identity, values, epistemology) by tracking the co-variation of these frame elements over time. Thus, EFT provides us with the basis for a process model of integrated learning, and ENA with the core of a method to trace the learning and development. How does ENA currently work? As described in Shaffer et al. (2009) and Rupp et al. (2009), the core idea is to apply methods of Social Network Analysis not, as would be usually the case, to networks of relations between people, but to relations between the elements of epistemic frames as they develop over time. In a first step, observations on learners (such as face to face communication, email messages, any other kind of data that can be interpreted in the SKIVE framework) are coded in terms of the SKIVE elements (currently by human raters). This content analysis yields the basic data structure: a sequence of observations recording these occurrences. Co-occurrences as the basic measure of strength of relationship between the SKIVE elements can be calculated for each time slice, and/or accumulated over time slices. In both cases, we get a SKIVE x SKIVE adjacency matrix which can be subjected to the various methods originally developed for Social Network Analysis (Wasserman & Faust, 1994). Using these adjacency matrices and appropriate algorithms, one can also visualize the relation between the matrix elements as (undirected) graphs. p. 41

Very interesting p. 41

The ENA method has so far been mainly validated in form of ethnographic studies that tested if the method can detect relations that can be seen by human observers (e.g., Hatfield & Shaffer, 2010) and in (quasi-) experimental settings. A study reported in Shaffer et al. (2009) revealed, for instance, that ENA measures were sensitive to conditions where a mentor was present to support students’ learning. In a number of projects, the conceptual and psychometric challenges involved in turning ENA into a rigorous assessment method are currently being tackled (see http://epistemicgames.org/eg/category/publications/). p. 42

A short term goal in the writing scenario is to make use of the CEFR assessment framework due to its immediate and obvious relevance to assessing language competencies. A mid-term goal (M12-18) is to also apply the Epistemic Network Analysis methodology to provide feedback on students writing. The basis for this is to use process writing (multiple versions of a student essay developing over weeks) as a means to trace developments on the SKIVE elements of ENA, as introduced above. This requires initial training for teachers in applying these dimensions to a students' writing corpus (maintained in the e-portfolio). Once teachers use the SKIVE elements reliably and continuously, we can again employ machine learning techniques to exploit the relations between teachers' (and peers') coding of students' writing products and the Identified SKIVE dimensions, over time turning teachers' manual coding into a software service. p. 44

For our initial work with language teachers, however, good enough speech-to-text technology will not be available. To nevertheless provide support for assessment we will employ an observational approach: While it is very difficult for teachers to monitor parallel groups, it is less of a challenge to analyse these conversation streams sequentially, using “video” recordings of each student group's virtual space. (The disadvantage is, of course, that feedback will be delayed.) p. 48

The designer and planner build upon the powerful ADOxx metamodelling platform enabling a rapid development and adaptation of the method and supporting requirements of NEXT-TELL on platform level p. 50

Three iterations of specification and development are planned during the runtime of the project. p. 53

Albert, D., Kickmeier-Rust, M. D., & Matsuda, F. (2008). A formal framework for modelling the developmental course of competence and performance in the distance, speed, and time domain. Developmental Review, 28, 401-420. Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman. Botturi, L., & Stubbs, T. (Eds.). (2007). Handbook of visual languages for instructional design: Theories and practice: IGI. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press, http://www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf Conole, G., & Fill, K. (2005). A learning design toolkit to create pedagogically effective learning activities. Journal of Interactive Multimedia Education, 8. Retrieved from www.jime.open.ac.uk/2005/08/ Cowie, B., & Bell, B. (1999), A model of formative assessment in science education, Assessment in Education, 6: 101-11. Crawford, V. M., Schlager, M. S., Penuel, W. R., & Toyama, Y. (2008). Supporting the art of teaching in a data- rich, high-performance learning environment. In E. B. Mandinach & M. Honey (Eds.), Data-driven school improvement (pp. 109-129). New York: Teachers College Press. Currier, S. L., Campbell, L., & Beetham, H. (2005). JISC pedagogical vocabularies project. Report 1: Pedagogical vocabularies report Available from http://www.jisc.ac.uk/uploaded_documents/PedVocab_VocabsReport_v0p11.doc Doignon, J., & Falmagne, J. (1985). Spaces for the assessment of knowledge. International Journal of Man- Machine Studies, 23, 175–196. Doignon, J., & Falmagne, J. (1999). Knowledge Spaces. Berlin: Springer. Falconer, I., Beetham, H., Oliver, R., Lockyer, L., & Littlejohn, A. (2007). Mod4L Final Report: Representing learning designs. Glasgow: Caledonian University. Hatfield, D., & Shaffer, D. W. (2010). The epistemography of a journalism practicum: The complex mechanisms of developing journalistic expertise (WCER Working Paper No. 2010-10). Retrieved February, 2011, from http://www.wcer.wisc.edu/publications/workingPapers/ papers.php Hymes, Dell. 1972. On communicative competence. In J. B. Pride and J. Holmes (eds.): Sosciolinguistics. Penguin 269-93. Jolliffe, F. (1991). Assessment of the Understanding of Statistical Concepts. In D. Vere-Jones, Proceedings of the Thirs International Conference on Teaching Statistics, Vol.1 (pp. 461-466). Otago, NZ: Otago University Press. Karagiannis, D.; Kühn, H. (2002): Metamodelling Platforms. Invited Paper. In: Bauknecht, K.; Min Tjoa, A.; Quirchmayer, G. (Eds.): Proceedings of the Third International Conference EC-Web 2002 – Dexa 2002, Aix-en-Provence, France, September 2-6, 2002. LNCS 2455, Springer-Verlag, Berlin, Heidelberg, p. 182. Karagiannis, D., Grossmann, W., & Hoefferer, P. (2008). Open Model Initiative. A feasibility study. Vienna: University of Vienna. Karagiannis, D. (2010). Conceptualization of Modelling Methods. Training Session/Training Material. November 22-24, 2010. Vienna. Koper, R. (2005). An introduction to learning design. In R. Koper & C. Tattersall (Eds.), Learning design. A handbook on modelling and delivering networked education and training (pp. 3-20). Berlin: Springer. Koper, R., & Tattersall, C. (Eds.). (2005). Learning design. A handbook on modelling and delivering networked education and training. Berlin: Springer. Korossy, K. (1997). Extending the Theory of Knowledge Spaces: A Competence–Performance Approach. Zeitschrift für Psychologie, 205, 53–82. © NEXT-TELL consortium: all rights reserved page 51 D2.1 Specification of ECAAD Methodology V1 p. 55

Korossy, K. (1999). Organizing and Controlling Learning Processes Within Competence–Performance Structures. In D. Albert & J. Lukas (Eds.), Knowledge Spaces: Theories, Empirical Research Applications (pp. 157– 178). Mahwah, NJ: Lawrence Erlbaum Associates. Law, N., Pelgrum, W. J., & Plomp, T. (2008). Pedagogy in ICT use (“Sites 2006”). Berlin: Springer. McMillan, J. H. (2003). Understanding and improving teachers' classroom assessment decision making: Implications for theory and practice. Educational Measurement: Issues and Practice, 34-43. McAndrew, P., Goodyear, P., & Dalziel, J. (2006). Patterns, designs and activities: unifying descriptions of learning structures. International Journal of Learning Technology, 2, 216-242. McMillan, J. H. (2003). Understanding and improving teachers' classroom assessment decision making: Implications for theory and practice. Educational Measurement: Issues and Practice, 34-43. Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing (PADI technical report 7). Stanford, CA: SRI. Mislevy, R. J., Steinberg, L., & Almond, R. G. (1999). Evidence-centered assessment design Available from http://www.education.umd.edu/EDMS/mislevy/papers/ECD_overview.html Morris, A. K., & Hiebert, J. (2011). Creating shared instructional products: An alternative approach to improving teaching. Educational Reseacher, 40(1), 5-14. National Academy of Sciences. (1996). National Science Education Standards. Washington, DC: National Academy Press. Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. Cambridge: Cambridge University Press. Pachler, N., Mellar, H., Daly, C., Mor, Y., & Dillan, W. (2009). Scoping a vision for formative e-assessment: a project report for JISC (April 2009). London: The WLE Centre, Institute of Education. Paquette, G. (2003). Instructional engineering for network-based learning: Pfeiffer/Wiley Publishing Co. Popham, W. J. (2008). Transformative assessment. Alexandria, VA: Association for Supervision and Curriculum Development. Riscontente, M. M., Mislevy, R. J., & Hamel, L. (2007). An introduction to PADI task templates. (Principled assessment designs for inquiry technical report 3). Rupp, A. A., Choi, Y., Gushta, M., Mislevy, R. J., & Thies, M. C. (2009). Modelling learning progressions in epistemic games with epistemic network analysis: principles for data analysis and generation. Madison, WS: University of Maryland. Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers and Education, 46(3), 223-234. Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nutly, A., Bagley, E., et al. (2009). Epistemic Network Analysis: A prototype of 21st-Century assessment of learning. International Journal of Learning and Media, 1(2), 33-53. Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153-189. Sloep, P., Hummel, H., & Manderveld, J. (2005). An introduction to learning design. In R. Koper & C. Tattersall (Eds.), Learning design. A handbook on modelling and delivering networked education and training (pp. 139-160). Berlin: Springer. Toulmin, S. E. (1958). The uses of argument. Wasserman, S., & Faust, K. (1994). Social Network Analysis: Methods and Applications. Cambridge, UK: Cambridge University Press. Warschauer, M., & Meskill, C. (2000). Technology and second language learning. In J. Rosenthal (Ed.), Handbook of undergraduate second language education (pp. 303-318). Mahwah, New Jersey: Lawrence Erlbaum. Wong, W. Y., & Reimann, P. (2009). Web based Educational Video Teaching and Learning platform with Collaborative Annotation. In I. Aedo, N.-S. Chen, Kinshuk, D. Sampson & L. Zaitseva (Eds.), The Ninth IEEE International Conference on Advanced Learning Technologiies (ICALT 2009) (pp. 696-700). Riga, Latvia: IEEE. Wood, R. (1968). Objectives in teaching of mathematics. Educational Research, 10, 83-98. p. 56

LEPP p. 57

Longitudinal Evaluation of Performance in Psychology (2nd generation e-portfolio) p. 57

PADI p. 57

The PADI project aims to provide a practical, theory-based approach to developing quality assessments of science inquiry by combining developments in cognitive psychology and research on science inquiry with advances in measurement theory and technology. p. 57

In the German speaking countries standards are comprehend as competences. The term goes back to Weinert (2001a), who defines specialized cognitive competences as “clusters of cognitive prerequisites that must be available for an individual to perform well in a particular area” (p.47). These cognitive competences require long-term learning, a broad experience, a deep understanding of the topic, and automatic action routines that  p. 59

must be controlled at a high level of awareness. Furthermore according to Weinert, competences do not only include cognitive abilities but also the motivation and willingness to apply knowledge in different situations. p. 60