Abstract Science Created by You (SCY) is a project on learning in science and technology domains. SCY uses a pedagogical approach that centres around products, called ‘emerging learn- ing objects’ (ELOs) that are created by students. Students work individually and collabo- ratively in SCY-Lab (the general SCY learning environment) on ‘missions’ that are guided by socio-scientific questions (for example ‘How can we design a CO2-friendly house?’). Fulfilling SCY missions requires a combination of knowledge from different content areas (eg, physics, mathematics, biology, as well as social sciences). While on a SCY mission, students perform several types of learning activities that can be charac- terised as productive processes (experiment, game, share, explain, design, etc), they encounter multiple resources, collaborate with varying coalitions of peers and use changing constellations of tools and scaffolds. The configuration of SCY-Lab is adaptive to the actual learning situation and may provide advice to students on appropriate learning activities, resources, tools and scaffolds, or peer students who can support the learning process. The SCY project aims at students between 12 and 18 years old. In the course of the project, a total of four SCY missions will be developed, of which one is currently available. p. 1

The Science Created by You (SCY) project intends to offer students a learning experience that is based on real life, challenging, assignments and that includes both inquiry and collaboration. SCY additionally offers a view of learning as a process of creation, leading to the production of so-called ‘emerging learning objects’ (ELOs; see Hoppe et al., 2005). These ELOs form the pivot of the SCY pedagogical approach. To create ELOs, students must gather and process information, design and conduct experiments, make interpretations and abstractions, and communicate their conclusions; in other words, they must engage in active learning processes. ELOs can be created by individual students, but also in collaboration between students. ELOs also play a role in providing students with adaptive support through so-called pedagogical agents. These agents analyse students’ progress based on the ELOs they produce, together with the log files of their interaction with SCY-Lab (the general SCY learning environment), and their communication with fellow students. p. 2

Learning by creating knowledge is one of the basic ideas behind constructionism: ‘…knowledge construction takes place when students are engaged in building objects.’ (Kafai & Resnick, 1996, p. 2). The objects constructed can be computer models (Hestenes, 1987; Pata & Sarapuu, 2006), physical objects and artefacts (Crismond, 2001), drawings (Hmelo, Holton & Kolodner, 2000), concept maps (Novak, 1990), computer programs (Mayer & Fay, 1987), podcasts (Lee, McLough- lin & Chan, 2008), experimental designs (Etkina et al., 2010), or even instruction (Vreman-de Olde & de Jong, 2006). p. 2

SCY’s ELOs include (System Dynamics) models, concept maps, artefacts, data sets, hypotheses, tables, summaries, reports, plans and lists of learning goals. All of these ELOs result from learn- ing activities. Partly based on work by Anderson & Kraftwohl (2001) and Mayer (2002), we have identified in SCY 53 different learning activities with associated ELOs. For example, one learning activity is ‘classification of examples’, for which the associated ELO is a ‘table with categories and examples’. So that students can perform activities and thus create ELOs, SCY-Lab (the SCY learning environment) provides them with dedicated tools for tasks such as modelling, concept mapping, writing reports, gathering data from simulations and analysing data tables. Tools can be adapted to the student or the context by supplying scaffolds, adaptations to the tools that inform or support students. One such scaffold is the provision of a partly worked-out system dynamics model, instead of letting students create a complete model from scratch. Tools are supplemented with services that help students in their work but do not lead directly to the production of ELOs. An example of a service is the awareness service that gives an overview of the presence and activities of peer students. In summary, students perform learning activities p. 2

with the help of (possibly scaffolded) tools that create ELOs and they use services for mainte- nance of the learning process. p. 3

To help students navigate through SCY-Lab, learning activities are grouped into so-called Learning Activity Spaces (LASs). A LAS is a combination of activities and ELOs that form a conceptual unit. Each LAS has one (or on some occasions more than one) central object(s) that is called the anchor ELO(s). An example of a LAS is ‘Experiment’. In this LAS, students need to collect data with learning activities such as ‘run experiment’ for which the resulting ELO is a ‘data set’ (which is the anchor ELO for this LAS). LASs in their turn are combined into different pedagogical scenarios. A SCY mission is a SCY learning environment (SCY-Lab) filled with content. Designers of a SCY mission can choose one specific scenario as the pedagogical theme for a mission; by doing so, they will know the LASs involved, the learning activities that are supported and ELOs that need to be produced. We have distinguished a total of 13 different scenarios, labelled ‘design challenge’, ‘problem resolution’, ‘close a case’, etc (for a complete overview see, Weinberger et al., 2009). As an example, Figure 1 displays the configuration of LASs for the scenario called ‘design challenge’. This scenario was the guiding scenario for the first SCY mission that has been developed. The overall topic of this first mission was the design of a CO2-friendly house. p. 3

To further illustrate this scenario we focus on one LAS from it, namely, ‘conceptualisation’. In this LAS, students try to identify the different concepts involved in the mission. For instance, students might try to find out what CO2 is, how it is produced, what types of activities in the house can produce it, what energy is and what the relation is between energy and CO2. The (anchor) ELOs produced are a concept map linking the different concepts together and, in a later stage, hypoth- eses that can be investigated. Students will return to ‘conceptualisation’ in the course of the scenario, for example, after they have done some reflection. The diamonds represent the ELOs produced (and consumed) by a specific LAS. In the next few sections, we focus on the different roles that ELOs play in SCY learning environments. p. 3

ELOs play a pivotal role in SCY. First, they form the central pedagogical concept; the entire pedagogical approach is organised around the principle that learning is seen as producing ELOs. Second, they form the basis for collaboration among students, both implicitly by reuse of existing ELOs created by other students and explicitly as shared objects on which multiple students work at the same time. Third, ELOs are a main input for automatic analysis of student progress by p. 3

pedagogical agents, leading to the shaping of the learning environment to the students’ indi- vidual characteristics. Fourth, ELOs form the basis for e-portfolios and student-centred assess- ment in SCY. In the preceding section, we discussed our pedagogy and the role of ELOs in it; the other aspects will be elaborated in the following subsections. p. 4

In SCY, collaboration may be explicit, when students work together synchronously on creating an object; or it may be more implicit when students reuse each other’s objects. p. 4

First, when collaboratively creating an ELO such as a concept map (with SCYMapper, the SCY concept-mapping tool) the view of the concept map is shared and all students involved are able to make changes to this concept map. To emphasise the centrality of the ELOs they are projected in the middle of the screen. Tools (such as SCYMapper) to edit ELOs are opened from the ELOs, and ELOs on which collaboration takes place are extended with ‘drawers’. One drawer contains infor- mation on peer students who are sharing the object; another drawer contains the chat (SCYchat). In this way, the chat is very close to the object to which it refers and chat conversation is saved with the ELO. Figure 2 shows a configuration of SCY-Lab with SCYMapper; SCYchat and avail- able peers are attached as drawers. The content comes from SCY Mission 1. Students are creating a concept map on CO2 emissions (other parts of this interface are explained when the first SCY mission is discussed later in the article). p. 4

Second (groups of) students may use ELOs created by other (groups of) students. The SCY project has developed an open learning object repository that is called RoOLO (Repository of Open Learning Objects). Students can search RoOLO to find appropriate learning objects from others (or created by themselves) that are saved there. Pedagogical agents (see the next section for a further description of agents) can also suggest ELOs from RoOLO for the student(s) to use. This use of ELOs has consequences for the indexing and retrieval of learning objects. In the absence of indexing through experts, learning object metadata must be derived from the learning situation. This means that we must use ‘on the fly’ meta tagging. p. 5

The following ‘user story’ describes how the reuse of ELOs may take place: Chris works on a mission within SCY. In this context, he creates an ELO, called ELO B.0. Chris decides that his ELO will be visible to all students (‘public’). The next day, he continues his work on that ELO and stores a new version (‘ELO B.1’). Only the recent version ELO B.1 will be visible to the public; Chris will also be able to browse the previous versions of his ELO. Meanwhile, Olivia has found Chris’ ELO B.1 and because it seems to be relevant to her work, she starts working with it. Olivia, who is not the original author of the ELO, stores a modified version as ELO E.0. On his next login, Chris will be notified of the new ELO that now branches from his ELO—he might be interested in the changes Olivia made or he even may want to begin collaborating with her. p. 5

This example can be used to illustrate a few aspects of our ELOs. First, we keep track of different ‘editions’ of the ELO in two ways, versioning and forking. In SCY, a new version of an ELO will not overwrite previous versions, but it will include a reference to its predecessor when it is added to the repository. A version of an ELO is created whenever an ELO has been changed. This is comparable to a ‘save’ operation in a common software application, with the difference that old versions are retained. A student may purposefully go back to previous versions, as when working on alternative solutions. When an ELO has been modified by a different student than its original author or when this option has been explicitly chosen, a fork is created as the start of a new version chain. A fork is comparable to a ‘save as’ operation. Second, to support the search for ELOs in RoOLO, metadata include simple information such as title, author and creation date, along with versioning/forking indices, a list of keywords, the definition of the format and information on the context (mission collaboration, relations to other ELOs). The format is divided into four categories: (1) technical format (eg, image/gif), (2) logical representation (eg, list, tables, sketch, photo, etc), (3) functional role (eg, data set, hypothesis, concept map, etc) and (4) related activity (the learning activity that is associated with the ELO). Metadata can be particularly used to improve search results, which is very important for the reusability aspect. Therefore, it is possible not only to search for an ELO with a specific name or another version or fork of a given ELO, but also to find similar ELOs with respect to the format types or keywords. A lexical search in the ELO content (if it is textual content) will be provided as well. p. 5

As indicated above, the fact that students generate learning objects means that we have to add metadata ‘on the fly’. For SCY, we distinguish three different approaches to metadata generation that are associated with certain types of fields or tags: (1) Some of the fields must be filled by the user, such as the title or, according to the history, the versioning/forking keys; (2) Other fields can be filled at creation time using the information available in the tool. This not only relates to formal features such as creation time, media types or user information, but also extends to knowledge of object types and content features explicit in the tool (eg, a data processing tool might ‘know’ that a certain time series of numerical data represents the temperature in a spe- cific location); (3) For externally created data objects or objects with semantics not captured by the tool (such as texts), we use information extraction and data mining techniques in connec- tion with mission-specific domain ontologies. Some fields like the keywords are calculated p. 5

by SCY by using data mining techniques in connection to a prefabricated, mission-specific, ontology. p. 6

ELOs underlying an adaptive learning environment Students in SCY missions are active, have their own responsibilities and are involved in a large set of high level learning activities, such as performing investigations, creating models and collabo- rating with others. Students are supported in performing these activities by tools. These tools may change according to the context of learning (eg, individual or collaborative) and/or characteris- tics of the student (eg, current domain knowledge, learning approach [eg, working according to a strategy or more randomly]). We have called this adaptation scaffolding. A necessary input for adaptation is diagnosis of the status of the student, for which SCY uses information from several sources, including activity log files, chats and ELOs. The analysis is done by pedagogical agents that continuously analyse the ELOs, chats and actions of students. Results of the analysis by the agents are presented to the student in the form of pedagogical interventions such as the activation of scaffolds. In SCY, we do not keep a persistent representation of a user (as is done in ‘traditional’ student modelling), but the status of a student is inferred at a specific point in time when necessary. The next ‘user story’ describes how ELO analysis (in this case of a concept map) helps to provide students with adaptive hints and may suggest collaboration among students. p. 6

Eric just started working on his second SCY mission called the climate-friendly house. He has seen the challenge and decides to use the simulation tool ‘SCYSim’. He starts entering numbers and tries to make sense of the outcome. After some time, a question mark starts to glow in one corner of the simulator window. Eric clicks on the question mark. A pop-up information window is displayed: ‘Read more about relevant resources before working with this tool’, including a list of appropriate resources on thermal energy. Eric follows this advice and scans a list of resources on thermal energy. He starts reading the resources. Eric is reading quite difficult texts. The text reading is accompanied with the task of making a concept map on the energy needs of a house. The first step requires Eric to highlight those sections in the explanatory texts that he considers most important. To support Eric in his concept-mapping activity, a pedagogical agent then extracts the most central concepts from the highlighted text sections and displays them in the concept-mapping tool called ‘SCY- Mapper’ automatically. Eric now can choose from this list of concepts in naming the con- cepts and relations that he adds to his concept map. When Eric is finished with this activity, he adds two more concepts to the map, which he did not find in the list but which he considers useful. Eric saves his concept map, indicating that he considers his job to be finished. Now a ‘graph-comparing agent’ becomes active in the background. This agent compares Eric’s concept map to a reference solution that is not visible to the students. The agent detects that some important concepts are still missing. A message appears that lists the missing concepts and suggests using the Web Browsing Tool to collect more resources related to the new concepts. The relevant concepts are extracted from highlighted text sections by first performing a statistical analysis on word co-occurrences in those texts. Then the extracted terms are checked against the ontology of mission-relevant terms to prevent the occurrence of false positives, ie, terms that are central in a highlighted text, but not related to the mission topics. p. 6

We use the graph edit distance (GED) algorithm to determine that Eric’s and the reference concept maps are partly disjoint. The GED is used to determine how close the student’s concept map is to a reference solution and also to find ELOs of other students that could complement the student’s concept map. This is described in the continuation of the user story below. p. 6

After reading two other texts and highlighting some sections, Eric adds some concepts and relations to his concept map about energy needs of a house. He feels that he has now covered the main topics and saves his work as an ELO. But the graph-comparing agent detects that the ‘Geometry of the house’ issue is still missing. Another student, Michael, has the same goal as Eric and also creates a concept map that has some important aspects, but also completely misses the ‘Geometry of the house’ issue. This is detected by the graph- comparing agent. In some aspects, Michael’s concept map is thus relatively similar to Eric’s. If the goal of the mission is to create a concept map similar to the reference solution, Michael would not be a smart choice for collaboration with Eric, because both of them have overlooked the same aspects. Susan, however, has created a concept map that takes the ‘Geometry of the house’ into account, and she would thus be an interesting student for Eric to talk to. The SCYMapper tool displays messages initiated by the graph-comparing agent that suggest a collaboration among Susan, Eric and Michael. p. 7

ELOs as the basis for student-centred assessment In SCY, we focus on two means of student-centred assessment: ELO-based portfolios and ‘playful’ peer assessment. An obvious role for e-portfolios is foreseen within SCY, with its central role for ELOs as a vehicle for and result of learning. Specific tools for composing e-portfolios based on ELOs will be developed and evaluated, emphasising the role of ELOs as a means for students to evaluate themselves and to present their knowledge and skills to the outside world. In portfolio assessment, the focus is on how ELOs created by the student represent the knowledge, embedded in the ELOs, and skills such as competencies for collaboration, design, and inquiry, which are acquired by students working in a SCY mission, thus providing summative assessment (see eg, Mason, Pegler & Weller, 2004). For SCY Mission 1, for example (on designing a CO2-friendly house), the knowl- edge and skills to be acquired are outlined and put into categories of general science skills, general social and presentation skills, general science concepts pertaining to physics (thermodynamics, electricity, energy, astronomy), biology and mathematics. These more specific knowledge types and skills can also be linked to one or several of the ELO types (eg, hypothesis list or concept map). Collections of ELOs created by students represent the knowledge or skill that they have acquired. The following user story describes how ELOs are selected and placed in an e-portfolio and how these ELOs and the e-portfolio as a collection of ELOs are assessed by teachers, peers and agents. p. 7

Maria and Peter are struggling with their concept map on energy. Although they realise their concept map is not finished, they do feel the need for some feedback and place their model in their SCY portfolio. By pressing the ‘assess now’ button, a software agent is activated, checking all variables and relations in their model and comparing it to a reference model. The agent generates a status report on the model’s completeness and correctness, which is automatically attached to their model. Maria and Peter continue working on their concept map on energy types, but after some time they decide that they would like to have some more feedback. They press the ‘assess now’ button again and now they are asked if they would like to participate in peer-based assessment. They agree and make their current concept map version public. The agent matches Maria and Peter with two other students, Olivier and Sylvia. The two groups, however, feel unsure about how to give feedback to each other and click on ‘help’. The agent provides them with two peer feedback examples, showing written feedback and smiley scoring (one with high and one low score). Then the students share concept maps and provide feedback to each other in the form of a written note and a smiley score. Based on the feedback, they both revise their concept maps and post final versions in their e-portfolios. Maria and Peter continue to make several other ELOs. They extract the variables from their concept map and display the data in a table. They make a hypothesis about energy use and test their hypothesis by examining energy flow in a system dynamics model. They also write a report documenting their testing procedure. p. 7

After having finished the SCY mission activities, the students mark their e-portfolio as ready for teacher assessment. A notification is sent to the teacher, who then can assess the products that show the students’ advancing knowledge and get information about their learning process (collaboration, feedback on ELOs, sequence of ELO production, etc). For the ‘are able to design, plan, and perform (virtual) experiments to these hypotheses’ skill, for example, there might be one preferred path of scientific inquiry that the students should follow, so that if a student actually follows that path, that shows that they actually posses this skill. The teacher assesses the e-portfolio content and provides the students with both grades and written feedback. p. 8

Shute (2007) argues for a unified approach to education—a new, integrated approach that uses the best of both summative (of learning) and formative (for learning) assessment by leveraging computer technology, educational measurement and cognitive science in order to know where the student started, where the student is headed, how the journey is progressing and the degree to which the destination is attained (Shute, 2007, p. 171). In SCY we adopt this unified approach and offer opportunities for summative and formative assessment through scaffolding, e-portfolios and peer assessment. p. 8

Students experience SCY through SCY-Lab. SCY-Lab can be filled with content, resulting in a SCY mission. The first mission that we developed is about designing a CO2-friendly (or CO2 neutral or climate-friendly house). CO2-friendly houses are houses that take environmental effects into account. The goal of this mission is to design such a house for one’s own circumstances. Students who work on this mission are 15–17-years old and work in a team of 3–4 students. They describe how CO2 emission from houses works and find out how this emission can be reduced. They consider materials, the design of the house, the way energy is supplied and used, and make a list of requirements and constraints for their house. The end product is a report with measurements and drawings of a CO2-friendly house. It is estimated that students will spend approximately 20 hours on this mission. The students work on computers through SCY-Lab, but they will also need to perform real experiments. ELOs that are produced in off-line work will be entered into SCY-Lab and then play their full ELO role. The scenario that has been used as the basis for this mission is the ‘design challenge’ as depicted in Figure 1. p. 8

SCY-Lab (Figure 3) contains several items: ELOs and resources, a mission map (bottom right), a search corner (bottom left) in which search actions can be performed and a planning tool (upper corner right). The set-up of SCY-Lab as presented in Figure 3 shows three collapsed ELOs and resources on the top left and one expanded resource in the centre. Resources are technically identical to ELOs, but they are provided to the student and not student generated. The ELO in the centre is the description of SCY Mission 1 ‘Design a CO2-friendly house’. It provides an introduc- tion to the mission, some context, and so on. The ELOs and resources at the top left are related to the mission start. The two ELOs with the arrow at the top left represent related anchor ELOs that are also represented in the mission map. At mission start, there are, of course, no ELOs created by the students. SCY-Lab provides templates for all ELOs that need to be produced. Templates can just be ‘empty’ ELOs, but it is also possible to provide a partially filled ELO as a starting point, such as a model that already contains the central variables. The mission map presents an overview of p. 8

anchor ELOs in the mission and helps students to navigate. Clicking on an anchor ELO opens the LAS that is related to this anchor ELO, and the anchor ELO is centred in the screen. New ELOs can be created or loaded through the toolbar buttons (see Figure 2). The button ‘New’ will reset the current tool with an empty ELO. ‘Load’ will pop up a dialogue in which selection of an ELO will open it. Save will save the current ELO, while at ‘Save as’ the ELO must be given a new name before saving it (creating a new fork). Students can also use the planning tool to create their own plan for traversing the mission; this allows them to sequence anchor ELOs according to their own preference. Collaboration on an ELO in SCY happens through shared screens augmented with a chat presented in a drawer attached to the right of the ELO. The persons to chat with appear in another drawer on top of the anchor ELO (see also Figure 2). Collaboration is initiated by drag- ging a person’s icon from the presence area on SCY-Lab over the ELO for which collaboration is needed. After exchanging confirmations, the collaboration is set up with the presence indicator and chat. The search facility in the bottom corner provides a keyword search to search for other students’ ELOs. Once a keyword is typed in, a search result box shows the search results in order of relevance. Students can then add ELOs from this result list to their workspace. Search is context sensitive. As it is known in what mission and what LAS the search is performed, only ELOs that have meaning in that context will be shown in the result list. SCY-Lab uses a familiar type of desktop metaphor with comparable features such as dragging, resizing, closing and so on. A specific feature is that windows can be rotated. This helps to create a real desktop metaphor that provides the look and feel of work in progress. Currently, SCY-Lab contains tools for creating concept maps, system dynamics models, text-based reports and designs for experiments, as well as interview schedules, a simulator that can generate data, and a tool to p. 9

organise and analyse both these data and data collected from other sources. SCY-Lab also provides services for chat-based communication and collaboration on the basis of a shared ELO. This means that students can collaborate on an ELO, in which case they will each see the ELO in a tool window and changes that one student makes will be propagated synchronously to collaborators. A chat is attached to such a shared ELO to allow for discussion of the work. Furthermore, SCY provides a tool for planning the work on the mission and for monitoring progress on the mission as a whole and for each individual ELO. p. 10

SCY has just entered its third year (out of four). The second year was marked by the release of SCY-Lab 1.0 and the first SCY mission as described in this paper. This release signifies the start of an extensive set of student evaluations, not only to see if the pedagogy works but also to investi- gate students’ reactions to an innovative interface and a complete new way of working. In the next paragraphs, we present the first results of such an evaluation study, but we will begin by presenting results from two pilot studies. p. 10

In preparing Mission 1, we conducted a number of pilot studies that used mock-ups of SCY-Lab or limited SCY-Lab functionality. Two of these studies investigated a central aspect of SCY, namely the use and reuse of ELOs. An initial pilot study took place in the Grenoble area (France). This study was conducted in two schools, with two classes of 18 students each (15 years old). Students produced ELOs in the context of the ‘design a CO2-friendly house’ mission. Students were divided into ‘expert groups’ and ‘design groups’. We found that there was little confrontation of the different possible solutions developed by the different groups. We conjecture that there were not sufficient constraints about students’ use of scientific explanations and argumentation to justify their choices. The analysis of level of argumentation produced by students in the ELOs showed that it seems necessary to give clearer assignments and constraints for the work in groups so that ELOs that are better justified scientifically will result. p. 10

A second pilot study took place in Amster- dam (the Netherlands). This study tried to find out whether and how students use the work of their peers to improve their own learning. The study focussed on nine fourth-graders who were engaged in the CO2-friendly house mission. At the start of this mission, students created a concept map to represent their prior knowledge about this topic. Students were twice given the opportu- nity to improve their concept maps on the basis of the knowledge acquired through mission work. On both occasions, they could consult a concept map that was created by a student from another school. Interpretative analyses point to three important findings. First, there is a large variety in the extent to which students use other-generated ELOs. Some students entirely disregarded other- generated ELOs as an additional learning resource, whereas others copied parts of other- generated ELOs and pasted them into their own ELOs. In initial stages of the task, only a few students frequently considered and continuously built on the other-generated ELO. Second, some students have principled objections against using other-generated ELOs. Building on work from peers is regarded as a socially undesirable, ‘unfair’ behaviour even when it is suggested to do so and when students are assured that no negative consequences should be expected when using other-generated ELOs. Moreover, some students seemed to work out their own approach to the task instead and did not always perceive the other-generated ELO as a useful resource. Third, students take critical stances against other-generated ELOs. Copy-paste behaviour implies that students take up parts of other-generated ELOs with little critical consideration of the adequacy of the ELO. However, none of the participants copied from the other-generated ELO without critically reflecting on its quality. p. 10

The first study with the complete SCY Mission 1 was conducted in Oslo (Norway). For four consecutive Wednesdays, 20 students (16 and 17 years old) worked to realise different designs of a CO2-friendly house. The total time for the project at school was 20 school hours (each 45 p. 10

minutes), but most of the groups also did homework in addition to the time at school. In par- ticular, the students used the house thermal simulation from the SCY mission. The simulation can calculate heat loss of elements in the building (different forms of insulation, building struc- ture, ventilation, windows, etc) depending on the parameters entered. This became their main tool throughout the trial. But the students were also encouraged to use information sources and tools outside SCY-Lab. The overall task, to design a CO2-friendly house, appeared to be a complicated task for the students. They found it conceptually difficult and seemed to experience an overall impression of SCY-Lab as complex. The investigations were overwhelming in the initial phase because students were confronted with a model for learning new to them and issues about which they did not know enough about to get started. SCY-Lab as a digital learning environment was new to them as well. Here, the approach of giving students ‘Learning activity spaces’ to navigate through the mission did not communicate well to the students in the current version of the technology. Yet, the task of actually drawing a house was a task they felt comfortable with and it seemed instrumental in moving the work ahead. The groups varied in their drawing activities, which ranged from drawing by hand, to using the drawing tool within SCY-Lab, to using Google SketchUp. Initial results indicate that the groups that used SketchUp got lost in the features of the tool, and their overall target for the project seemed to drift some- what towards making a nice and elegant house rather than a low emission house, whereas the groups using simpler tools kept better to their overall target. The students spent considerable time with the thermal simulator of the house to understand the way it worked and what they could do with it. Initial results from the observation of simulation use indicate that this use if time was task relevant, as students seemed to get a picture of building structures, types of insulation, importance of triple window frames versus heat exchangers in the ventilation and more. Yet, deeper analysis is needed to clarify this initial observation. A firmer result is that exchanging ELOs was a positive process for the students. At a particular point in the project, the students were asked to look up the simulations (data set ELOs) from other groups and study their approach and choices. This appears to have been entirely a positive contribution, occur- ring at this point in the study. The groups that were somewhat passive and lagging behind the others got new ideas; some groups seemed to get their approach confirmed by other groups that selected similarly, but also got into (internal) discussions in the group about the rationale for their choices, leading to reconsideration or to a firmer grounding for their choices. The positive result of ELO exchange is significant for SCY, as this is at the core of the research approach in the project. For instance, important further investigations of the first trial and later trials can relate to the point in time in the mission at which the exchange of ELOs should take place. In this study, the exchange was imposed on the students and was done after they had done con- siderable work with the simulation from scratch. Other issues for further analysis and study can relate to how students select ELOs, which types of ELOs are suitable (visual vs. textual, com- plicated vs. simple), and how they are used. p. 11

The upcoming further evaluation studies in SCY will therefore focus on the aspects of ELOs that have been central in this paper: ELOs as essential to the learning process, ELOs as sharable and reusable objects, and ELOs as a source for e-portfolios and peer assessment. An aspect that we will be able to develop further when students begin to use SCY missions is the automatic meta- tagging of ELOs. This is required for the functioning of SCY-Lab as a learning environment and for indexing ELOs in RoOLO to support retrieval and reuse. For the latter, we will particularly study the practical potential of these mechanisms in the sense of ‘affordances’ for knowledge sharing and exchange. An important issue here is that, beyond defining similarity measures for the learning objects, we need to validate that the technical similarity measure actually corre- sponds to what would be judged as similar from a user/task perspective. A study of this type has recently been conducted with educational design artefacts in the SCY context (Wichmann, p. 11

A similar study with ELOs will be needed to corroborate the practical benefits for knowledge sharing in the SCY student community. One aspect of SCY that has not yet been discussed in this paper and that is currently under development is the cockpit view of the learning process we intend to make available for teachers. This cockpit view enables teacher to gain insight into the learning process and forms the basis for decisions on tuning the func- tioning of pedagogical agents during the learning process. p. 12

References Anderson, L. W. & Kraftwohl, D. R. (Eds) (2001). A taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives. New York: Longman. Black, P. & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy, and Practice, 5, 7–74. Bransford, J. D., Brown, A. L. & Cocking, R. R. (Eds) (1999). How people learn: brain, mind, experience, and school. Washington, D.C: National Academy Press. Crismond, D. (2001). Learning and using science ideas when doing investigate-and-redesign tasks: a study of naive, novice, and expert designers doing constrained and scaffolded design work. Journal of Research in Science Teaching, 38, 791–820. Etkina, E., Karelina, A., Ruibal-Villasenor, M., Rosengrant, D., Jordan, R. & Hmelo-Silver, C. E. (2010). Design and reflection help students develop scientific abilities: learning in introductory physics laborato- ries. Journal of the Learning Sciences, 19, 54–98. Eysink, T. H. S., de Jong, T., Berthold, K., Kollöffel, B., Opfermann, M. & Wouters, P. (2009). Learner performance in multimedia learning arrangements: an analysis across instructional approaches. Ameri- can Educational Research Journal, 46, 1107–1149. Gijlers, H. & de Jong, T. (2009). Sharing and confronting propositions in collaborative inquiry learning. Cognition and Instruction, 27, 239–268. Hestenes, D. (1987). Towards a modeling theory of physics instruction. American Journal of Physics, 55, 440–454. Hickey, D. T., Kindfield, A. C. H., Horwitz, P. & Christie, M. A. (2003). Integrating curriculum, instruction, assessment, and evaluation in a technology-supported genetics environment. American Educational Research Journal, 40, 495–538. Hmelo, C. E., Holton, D. L. & Kolodner, J. L. (2000). Designing to learn about complex systems. Journal of the Learning Sciences, 9, 247–298. Hoppe, H. U., Pinkwart, N., Oelinger, M., Zeini, S., Verdejo, F., Barros, B. et al. (2005). Building bridges within learning communities through ontologies and ‘thematic objects’. In Proceedings of the 2005 Conference on Computer Support for Collaborative Learning. (pp. 211–220). Mahwah, NJ: Lawrence Erlbaum Associates. Jonassen, D. H. (1991). Objectivism versus constructivism: do we need a new philosophical paradigm? Educational Technology Research and Development, 39, 5–14. de Jong, T. (2006). Computer simulations – technological advances in inquiry learning. Science, 312, 532–533. Kafai, Y. B. & Resnick, M. (Eds) (1996). Constructionism in practice: designing, thinking, and learning in a digital world. Mawhaw, NJ: Lawrence Erlbaum Associates. Lee, M. J. W., McLoughlin, C. & Chan, A. (2008). Talk the talk: learner-generated podcasts as catalysts for knowledge creation. British Journal of Educational Technology, 39, 501–521. Linn, M. C., Lee, H.-S., Tinker, R., Husic, F. & Chiu, J. L. (2006). Teaching and assessing knowledge integra- tion in science. Science, 313, 1049–1050. Lou, Y. (2004). Understanding process and affective factors in small group versus individual learning with technology. Journal of Educational Computing Research, 31, 337–369. Lou, Y., Abrami, P. C. & d’Apollonia, S. (2001). Small group and individual learning with technology: a meta-analysis. Review of Educational Research, 71, 449–521. p. 12

Mason, R., Pegler, C. & Weller, M. (2004). E-portfolios: an assessment tool for online courses. British Journal of Educational Technology, 35, 717–727. Mayer, R. E. (2002). Rote versus meaningful learning. Theory into Practice, 41, 226–232. Mayer, R. E. & Fay, A. L. (1987). A chain of cognitive changes with learning to program in logo. Journal of Educational Psychology, 79, 269–279. Minner, D. D., Levy, A. J. & Century, J. (2010). Inquiry-based science instruction – what is it and does it matter? Results from a research synthesis years 1984 to 2002. Journal of Research in Science Teaching, 47, 474–496. Novak, J. D. (1990). Concept mapping: a useful tool for science education. Journal of Research in Science Teaching, 27, 937–949. Pata, K. & Sarapuu, T. (2006). A comparison of reasoning processes in a collaborative modelling environment: learning about genetics problems using virtual chat. International Journal of Science Education, 28, 1341–1368. Shute, V. J. (2007). Tensions, trends, tools, and technologies: time for an educational sea change. In C. A. Dwyer (Ed.), The future of assessment: shaping teaching and learning (pp. 139–187). New York: Taylor & Francis. Shute, V. J. (2009). Simply assessment. International Journal of Learning and Media, 1, 1–11. Vreman-de Olde, C. & de Jong, T. (2006). Scaffolding the design of assignments for a computer simulation. Journal of Computer Assisted Learning, 22, 63–74. Weinberger, A., Dolonen, J., Hovardas, A., Pedaste, M., Kluge, A., Ludvigsen, S. et al. (2009). SCY scenario handbook and pedagogical plans, version 1. Enschede: University of Twente. Wichmann, A., Engler, J. & Hoppe, H. U. (2010). Sharing educational scenario designs in practitioner communities. Paper presented at the ICLS, Chicago (USA). p. 13