Learner-Centered Evaluation of an Augmented Reality System for Embedded Engineering Education

— This paper presents an ongoing work on a multisensory Augmented Reality system for engineering education. The system, composed of both software and hardware elements, will be used in embedded electronics courses. We first present similar systems and their potential advantages for education. Then, we discuss the originality of our system. Finally, we present the general approach as well as a number of methods that we will use for the future learner-centered evaluation of the proposed AR system in realistic settings.


I. INTRODUCTION
HE term "Augmented Reality" (AR) was introduced in the early 1990s [1] to designate a specific form of Human-Computer Interaction (HCI), in which views of the real world are enhanced by computer-generated content [2].The real and virtual elements in an AR system are semantically and spatially related.Compared to Virtual Reality (VR), AR does not aim at representing the real world by a realistic virtual analogy.It aims at promoting "intuitive" and natural multimodal interaction [3].In addition to 2D and 3D computer-generated visual content, spatial audio, tactile and even olfactory stimulations can be incorporated to enhance the user's perception of the real world.T As underlined by [4], AR offers new possibilities in education.These authors, together with others, cite the following major advantages of using AR in education: • a possibility of presenting information "just-intime" and "just-in-place", which will reduce information search, error-likelihood and will enhance memorization and recall (e.g.[5], [6], [7]); • a possibility of visualizing complex relationships and abstract concepts ( [8], [9]); • a possibility of experiencing phenomena which are unlikely to be experienced in the real world ([10], [7]); • a possibility of "learning-by-doing" (i.e. of constructing knowledge actively and autonomously, [11], [12]); • a possibility of improving learners' motivation because of the enthusiasm when interacting with new technologies ( [13]).All this benefits are transposable to engineering education, in general, and to embedded electronics courses, in particular.
There are only few AR reality prototypes for engineering education.Some of them will be presented below.

II. AR IN ENGINEERING EDUCATION
Kaufmann and Schmalstieg [14] developed an AR system for mathematics and geometry education (Fig. 1).
Fig. 1 AR system for geometry education (adapted from [14]) The system is a 3D geometric construction tool for the improvement of spatial abilities and for the maximization of transfer in real settings.This system has not been formally evaluated in a real course.However, an informal evaluation showed that students were motivated to use it and did not need a long familiarization before using it in practice.Several problems such as eye-hand coordination without haptic feedback and fatigue were also pointed out.As for the possible applications of the system, students mentioned interactive conic sections, vector analysis, intersection problems, and elementary geometry.
Another example is the use of tangible interfaces (i.e.physical objects coupled to digital information) and AR models in engineering graphics courses to help students better understand the relationship between 3D objects and their projections [15].This system was tested with 35 engineering-major students.The study showed that the tangible interfaces significantly enhanced students' learning performance and their abilities to transfer 3D objects onto 2D projections.There was also high engagement with the AR models during the learning process.
AR was also used for teaching embedded electronics courses.When learning electronics, especially embedded systems, students have to face the challenge of understanding the mechanisms of several devices without actually seeing those interactions and functions.Even in laboratory practices with electronic boards, they can only manipulate them through the available inputs and outputs, whilst the operations happening inside the components remain invisible.Consequently, students do not always get to fully understand the studied concepts.
The augmented reality aims to overcome those obstacles in the learning process, especially in the early stages of Computer Engineering studies.Thus, Müller et al. [16] and Andujar et al. [17] proposed an AR system for the improvement of students' interactions with remote laboratories.
In [17], the use case is the design of a digital control system based on an FPGA development board.In this case, AR is used in order to give the user the sensation that certain lab functions can be handled just as they would be in the real laboratory itself.The authors designed the system with the aim of limiting students' possible discouragement due to the lack of physical contact.The system was evaluated with 36 students and 10 teachers.The results, both for students and teachers, showed improved learnability of the theoretical concepts taught in the different courses, high engagement and higher motivation to learn than with traditional methods.
The existing AR prototypes for embedded electronics education are mainly based on visual interaction.In the E2LP project [19], we tried to go beyond visual aspects only.We designed a multisensory AR system for electronics education.This system is briefly presented below.

III. THE E2LP AUGMENTED REALITY SYSTEM
The E2LP system consists of a camera which captures a video of an electronics board.It displays this video on a touchscreen positioned on a support inspired by the structure of an electronic lamp (Fig. 2).The system also comprises a tactile pointer, which indicates its position on the board and with that information and the information from the camera, the AR software displays the corresponding visual enhancement on the real view presented on the tactile screen.
With the help of the camera and the pointer, students can access the specifications of the board components they want to work on.From a pedagogical point of view, the AR software is structured in following three levels: • Exercises: These are basic tasks and, in the same time, perfect use cases for AR.They have a well-determined path to resolution and a solution which can directly be superposed on the hardware elements in a visual and animated manner.Exercises are also the first tasks students have to resolve when starting a new course.Therefore, AR can be a valuable tool for facilitation understanding and engagement.
• Problems: These are open-ended tasks which can have different solutions or solving methods.Compared to exercises, they are more challenging from the point of view of the information to be displayed by the AR system, as it can be different in the different solutions.However, the AR system can display "clues" to facilitate the resolution of the task.
• Projects: These challenging tasks require that students define by themselves both the objectives and the resources to be used for the development of the project.Thus, there is no pre-established path or solution to follow.In this case, the AR software could provide information about the general resources students can make use of.
The E2LP AR system will be evaluated with students and teachers, in real courses.The dimensions to be evaluated and the general approach which we will adopt are presented below.

IV. LEARNER-CENTRED EVALUATION OF THE E2LP AUGMENTED REALITY SYSTEM
We advocate an iterative and learner-centered design approach where learners and teachers are involved all through the design process and where designs and prototypes of the future pedagogical tools are tested iteratively.This learner-centered approach is based on: • paying particular attention to diverse learners' needs and characteristics, • providing a setting for fostering authenticity and inclusion, • encouraging openness to experience and personal growth, and • encouraging co-creation of knowledge.The main goal of this approach is to facilitate learners in becoming active, self-directed and self-responsible participants in the learning process, in which peers and the instructor serve as facilitators, motivators and personal resources.From a teaching and design perspective, this is a 32 PROCEEDINGS OF THE E2LP WORKSHOP, WARSAW, 2014 very complex, demanding endeavor.As an entry aid, the following four guiding principles of the Alliance for Excellent Education could be used [17]: • Learning is rigorous and based on career-ready expectations.• Learning is personalized.
• Learning is collaborative, relevant and applied.
• Learning is flexible, with learning taking place anytime, anywhere.Based on this general approach, we can use the following more concrete measure of the effectiveness of AR for students and teachers.

A. Measures of technological effectiveness and reliability
These measures will be based on the performance of the AR software and hardware, as well as on the performance of the AR system as whole.We will use a number of wellestablished indicators of technological effectiveness and reliability such as: • the number and variety of functions and exercises supported by the AR system, • its power consumption, • the real time behavior of events and expected actions, • its reliability, • its interoperability.The cost of each AR component and the cost of the integrated system will also be an important measure of the effectiveness of the final E2LP AR platform.Even though all these measures do not directly stem from the learnercentered approach, they may have an effect on teachers' and learners' first impressions of the system and, consequently, on its further adoption.
Another group of measures will concern the ease-of-use and the user experience with the AR system.

B. Measures of ease-of-use and user experience
These characteristics will be measured using both quantitative and qualitative measures, which will be applied first to the different components of the system (i.e.software, tracking, tactile pointer) and then, to the system as a whole.The quantitative measures will include but will not be limited to: • the success rate on different learning tasks relevant for the use of the system; • the time that a given task requires for its execution; • the error rate; • the time spent for recovering errors; • the rate to learners' and teachers' satisfaction with the AR system; • the number and the reasons for eventual rejections of the system during use, if any.These measures will be complemented by qualitative measures of user experience.These qualitative measures will include, but will not be limited to: • the nature of the learning tasks supported by the AR system.This metric will be based on the taxonomy of exercises, problems and projects presented earlier in this document; • the nature and types of the errors typically made when using the system; • the strategies of error recovery; • the typical and personal strategies of using the AR systems in different educational contexts; • the learners' and teachers values associated with the use of the system; • the learners' and teachers most important emotions when using the system (e.g.enjoyment, interest, frustration, curiosity, immersion, etc.); • their motivation before, during and after use; • the confidence in the system; • the learners' and teachers' comments on its efficiency, utility, acceptability and further adoption.The quantitative and qualitative measures of the ease of use and user experience with the AR system will be first evaluated within the framework of formative usability evaluations.Formative evaluation is a type of usability evaluation that helps to "form" the design for a product or service.Formative evaluations involve evaluating a product or service, usually with small user samples, during development, often iteratively, with the goal of detecting and eliminating usability problems.One important aspect of formative evaluation is that the audience for the observations and recommendations is the project team itself, used to immediately improve the design of the product or service and refine the development specifications.Results are usually less formal than in final or "summative" evaluation, as suits the needs of designers, developers, project managers, and other project participants.
The above-mentioned system effectiveness metrics will be then used in the final summative evaluation of the AR system.Summative evaluation will be done on the complete or near-complete platform under realistic conditions.The objective will be to determine if AR improves learning, collaboration and creativity.
This summative evaluation of the usability of the AR system is closely related to the evaluation of the AR system in real educational settings.

C. Measures of adoption of AR in real educational settings
Specific measures for the adoption of the AR system in real education settings will be used.The quantitative measures of adoption will include: • the number of laboratory exercises, problems, interdisciplinary projects created by AR; • the number of AR systems installed in universities; • the number of exercises, problems and projects created by instructors and students.
As for the qualitative measures of teachers' and learners' adoption, they will be based on the Levels Of Technology Implementation (LoTi) framework proposed by [18].The LoTi framework is based on seven discrete implementation levels ranging from Nonuse (Level 0) to Refinement (Level 6).A synthesis of the LoTi framework is presented below: • Nonuse: a perceived lack of access to technology-based tools or a lack of time to pursue electronic technology implementation.

•
Awareness: the use of the proposed technological system is generally one step further compared to level 0. However, they still have little or no relevance to the individual teacher's instructional program.• Refinement: technology is perceived as a tool to help students solve authentic problems related to an identified real-world problem or issue.Even though we do not expect to see high levels of AR adoption because of the prototypical nature of the current system, we would like to apply this evaluation framework because of its prospective orientation.

V. CONCLUSION
We presented an AR system for embedded electronics education developed within the framework of the E2LP project.A number of measures for evaluating its effectiveness were also presented in this paper.The use of these measures will be based on a learner-centered approach.The results of the evaluation will be presented in the near future.

Fig. 2 .
Fig. 2. E2LP AR system: general view and view of the student's screen • Exploration:technology-based tools serve as a supplement to existing instructional program.The new technology is employed either as extension activities or as enrichment exercises to the instructional program.• Infusion: technology-based tools augment isolated instructional events.• Integration: technology-based tools are integrated in a manner to provide a rich context for students' understanding of the pertinent concepts, themes, and processes.Technology is perceived as a tool to identify and solve authentic problems relating to an overall theme/concept.• Expansion: technology access is extended beyond the classroom.