Future and Backward Exploration of XR Environments

XR environments are successfully used in various domains, such as medicine, education, training, and industry. Such environments contain domain knowledge expressed through content and users' behavior. However, current approaches to XR creation lack possibility of exploration of the knowledge included in the environments and expressed by the users' behavior. In this paper, we propose a method of knowledge exploration in XR environments, enabling the analysis of past and potential behavior of users and objects, with queries and automated reasoning. This solution aims to enhance knowledge dissemination using XR.


I. INTRODUCTION
XR SYSTEMS' rising popularity stems from their po- tential across domains and their ability to offer immersive experiences.However, existing systems often overlook event and interaction analysis within virtual environments, leading to a loss of valuable information.To enhance user experience, there's a crucial need to focus on exploring and analyzing XR system data for hidden insights.
Actions and interactions within XR environments can be explored through semantic queries and automated reasoning, particularly beneficial in employee training systems.This exploration offers valuable insights into users' behavior across different timeframes.
The knowledge gained from exploration is vital for monitoring, analyzing, and controlling XR environments, as well as understanding users' skills, experiences, interests, and preferences.Domain-specific terminology helps specialists, the primary users of XR environments, take full advantage of behavior exploration.
This paper introduces a new method for exploring XR environments, encompassing both forward and backward exploration through semantic queries on the XR environment's knowledge-based representation.The proposed methods were applied to represent the behavior of the virtual environment in an industrial XR training system developed for Amica S.A, Poland's leading house appliances manufacturer, to train employees on specialized industrial devices.
The paper's structure is as follows: Section II presents an overview of the current state of exploring virtual environments.Then, Section III explains the methods used to describe virtual content.Section IV elaborates on exploration methods with examples and an overview of exploration for the XR training system.Finally, Section V concludes the paper and suggests potential areas for future research.

II. RELATED WORK
Numerous approaches model 3D content behavior using ontologies and semantic web standards.One notable approach, as discussed in Pellens et al. [1], [2], [3], introduces temporal operators for expressing both primitive and complex behaviors.They also offer a graphical tool to model complex behavior using diagrams, encoding it within X3D scenes [4].
In another approach, De Troyer et al. [5] combine primitive actions like move, turn, and rotate to represent complex behavior in a user-friendly manner.This approach enables end users to specify complex behavior without extensive knowledge of 3D graphics and animation.
Krieg-Brückner et al. [6] introduce a tool using semantic concepts, services, and hybrid automata to describe 3D content behavior.The tool consists of a client component based on a 3D content presentation tool (e.g., XML3D browser) and a server component with various services for content selection and configuration.Additionally, an extra module manages intelligent avatars and their perception of the scene.
Chmiel et al. [7] proposed XSD-based semantic metadata schemes for 3D object interactivity, specifying events, conditions, and actions.CL ontologies [8], [9] represent multiuser virtual environments and avatars, defining geometry, space, animation, and behavior of 3D content.They include semantic counterparts to widely used formats like VRML and X3D.Environmental objects are described by attributes like translation, rotation, and scale, while avatars have names, statuses, user interfaces (UIs), and behavior defined through code bases In recent research, there has been a notable focus on humans and their interactions within virtual reality, particularly in creating ontologies for experimental purposes.One such recent work is the Virtual Human-Building Interaction Experimentation Ontology (VHBIEO) proposed by Chokwitthaya et al. [10].The primary objective of this solution is to establish a standardized approach for conducting experiments related to human-building interactions within virtual reality environments.In a similar vein, Heitmayer et al. [11] have introduced another ontology focusing on the human-centred analysis and design of virtual reality conferencing.The main goals of this ontology include enhancing user experience, facilitating research on VR conferencing (particularly in the realms of psychology and behavior), and enabling the sharing of research findings among the scientific community.
Proceedings of the 18 th Conference on Computer Science and Intelligence Systems pp.961-965 III.REPRESENTATION OF XR ENVIRONMENT In order for the XR environment to be appropriate for exploration, it needs a proper representation of interaction and 3D content.This can be created using knowledge representation technologies such as the semantic web (the Resource Description Framework-RDF [12], the RDF Schema-RDFS [13], the Web Ontology Language-OWL [14] and the SPARQL query language [15]) and ontologies.
For the presented methods of exploration, we prepared behavioral semantic models for the representation of activities, called Activity Ontology, as well as the representation of the workflow.

A. Representation of Activities and Features
Activities and features act as domain-specific states that connect the behavior model and the implementations of XR components.Within the Activity Ontology, the Semantic Web approach defines the representation of these activities and properties.This ontology comprises a TBox and an RBox that contain axioms outlining how users' and objects' features and activities are implemented within XR components and environments.The activity ontology is well suited for both procedural and object-oriented XR implementations.
Fig. 1 shows how class in the XR system is mapped to ontology to activity ontology.Since both procedural and object-oriented XR implementations rely on functions as the fundamental building blocks of workflow, the activity ontology supports structuring both types of implementations.
The ontology specifies the following classes and attributes associated with code elements: 1) ApplicationClass -is the class of all application classes specified in the code of XR components.2) Variable -is the class of all variables.
3) ParameterList -is the class of all lists of method parameters.4) ClassMethod -is the class of all methods specified in the code of XR components The example of a knowledge base generated using activity ontology is presented in listing 1.The example shows described class Battery, which has variable Charge and method ConnectRectifier.
Listing 1.A fragment of knowledge base describing element of XR system ao:Battery rdf:type owl:NamedIndividual , ao:ApplicationClass .

B. Workflow Representation
The workflow representation adapts the behavior model to describe states, events, and time related to the execution of class methods.Hence, it enables the specification of XR components' behavior upon their underlying imperative implementation.The example of the mapping method of the XR system to behavior model ontology is presented in Fig. 2.An event mapping states that the beginning of an activity is a method invocation or that a method completion is the finish of an activity.Hence, event mappings enclose states of method executions using domain events.Using such mapping, the system can generate a knowledge base, which can be treated as behavioral logs, therefore, can be explored by proper queries.An example of such a knowledge base is presented in Listing 2. The example describes two events that happened while using the system.The first event was "The visual inspection of the battery".That event began BatteryState1 and finished BatteryState2.This event had assigned TimeSlice, which gives us information about when and how long the event lasted.IV.EXPLORATION In this section, we propose different types of knowledge exploration and visualization within explorable XR environments, utilizing the models proposed in the previous section.We classify these exploration types based on the target periods, distinguishing between simulation with forward and backward exploration.
Simulation and forward exploration facilitate the process of reasoning and querying potential events and states in the XR environment.These events and states may remain undetermined, contingent upon the occurrence or absence of other events and states within the environment.Engaging in simulation and forward exploration necessitates a composed XR environment, which does not necessarily need to be compiled and run.Consequently, simulation and forward exploration can be initiated promptly after composing the environment.
Backward exploration enables the process of reasoning, querying, and visualizing past and present events and states that have been logged in behavior records.To engage in backward exploration, the XR environment must be executed, and behavior logs need to be generated.
Queries play a crucial role in acquiring knowledge about explorable XR environments.Nevertheless, despite their possible support in query languages such as SPARQL, our focus does not revolve around query result presentation operations like result limitation, sorting, or data aggregation.Instead, our primary emphasis lies in utilizing queries to gain deep insights into the properties and behaviors of the XR environment, facilitating exploration and understanding.
By applying these techniques for knowledge exploration and visualization, we elevate our comprehension of the dynamic nature of XR environments and empower users to interact effectively, reason, and query within these environments.These approaches serve as invaluable tools for researchers and developers, enabling them to delve into the potential of explorable XR environments and propel innovation in this field.
We used new exploration methods to query the behavior of users and 3D objects inside a virtual environment for the industrial worker training XR system, which allows trainees to learn how to act safely in an industrial setting.The training scenario implemented in the system focuses on safe work with a forklift.It was developed using resources from Amica S.A., a major producer of household equipment in Poland.

A. Simulation with forward exploration
Forward exploration facilitates reasoning and queries about the potential behaviors of users and objects within explorable XR environments.It encompasses various states and events associated with features and activities, encompassing autonomous actions and interactions among users, objects, and their interplay.Crucially, forward exploration is intimately intertwined with the simulation of environmental behavior, which aims to fulfil conditions necessary for events and states.As a result, simulation precedes forward exploration, encompassing both aspects within simulation queries.Importantly, since simulation and forward exploration revolve around potential events and states, they necessitate a workflow specification but do not mandate that the XR environment be actively running.
Simulation queries to an environment specification enable forward exploration of the environment without running it.Therefore, they must specify the conditions for which the exploration is accomplished.The illustrative simulation queries presented in this section assume that the delay between an event and another following event is equal to 0.001, and no exceptions are thrown during the execution of methods: Delay =0.001 exception(executed(M ethod, ExecutionID), null).
To allow forward exploration, the following elements of the XR system were mapped to semantic representation using Activity Ontology: Classes: 1) Trainee (class representing trainee), 2) Forklift (class representing forklift), 3) Battery (class representing battery), 4) Rectifier (class representing rectifier).Methods: 1) startForkliftInspection (method of Trainee class), 2) finishForkliftInspection (method of Trainee class), 3) chargeBattery(method of Rectifier class), 4) insertBattery(method of Forklift class), 5) showChargingState(method of Rectifier class), 6) plugIn(method of Rectifier class) Additional predicates were also used, allowing for the representation of temporal entities: time(event,tp) -predicate that is true for a given event and a time point if and only if the event occurs at the time point.
holds(event,ti) -predicate that is true for a given event and a time interval if and only if the fluent is true within the time interval Using such prepared knowledge representation, we are able to create the following queries: 1. How long will it take to check the visual state of a forklift?time(startF orklif tInspection(T rainee, F orklif t)), T P start ) time(f inishF orkLif tInspection, T P end ) Lenght = T P end − T P start It determines the time points of starting and finishing a forklift inspection and calculates the inspection length.(The Fig. 3 presents how a user inspects the forklift in XR system.)The query result is the following: T P start = 10, T P end = 22, Lenght = 12 2. What will happen after a trainee finishes charging the battery?holds(chargeBattery(T rainee, Battery), T I 1 ) holds(Action, T I 2 ) af ter(T 1 , T 2 ) Unlike forward exploration, which relies on simulating the environment's behavior, backward exploration uses logged activities.This eliminates the need for environmental behavior simulation.Furthermore, the inclusion of temporal statements with visual descriptors in behavior logs enables the visualization of past activities.
Queries specifically designed for backward exploration are referred to as exploration queries.The output of an exploration query is defined similarly to a simulation query.The behavior logs are structured based on RDF, enabling the utilization of the SPARQL language for conducting backward exploration.The example of behavior logs in the form of knowledge-base was presented in Listing 2.
PushButtonEvent .?PushButtonTimeSlice fo:timePoint ?pushButtonTime .FILTER ( ?time < ?pushButtonTime)}ORDER BY ?time The first query provides information about what happened in the scene before the trainee pushed the press button, which activated the forklift.The query searches for events and their assigned time slices that happened before the event with the name "Turning on the forklift".The action of turning on the forklift presents Fig. 4. The second query gives information about the duration of the visual inspection of the battery.The query searches for the time slice of the event named "visually controlling the battery", then using the time interval object, access the information when the event started and ended.Fig. 5  The last query provides information about what happened sequentially with the battery.The query searches for events that changed the states of the scene object named "battery".Then using proper time slices, the query determines the time by which the states are ordered.The results consist of the name and id of the states and the time when they have begun.

V. CONCLUSIONS AND FUTURE WORK
The use of exploration of behavior and 3D content in XR systems can have multiple applications, like learning about users, their experience, preferences, and interests and measuring their skills.This can be beneficial, for example, in virtual training, where we want to maximize the training results.
In this paper, we have proposed methods for forward and backward exploration based on semantic queries.The presented approach uses the knowledge-based representation of the XR environment, which is represented by described ontologies.Moreover, we provided examples of explorations based on the developed XR system for employee training in an industrial environment.
The possible future research directions could focus on developing available inexperienced user plug-ins for simplifying and automating the process of creating semantic queries and visualization of the results.

Fig. 1 .
Fig. 1.Mapping XR system class to Activity Ontology

Fig. 2 .
Fig. 2. Mapping the XR system method to fluent ontology