Simulation goals and metrics identification

Agent-Based Modeling and Simulation (ABMS) is a very useful means for producing high quality models during simulation studies. When ABMS is part of a methodological approach it becomes important to have a method for identifying the objectives of the simulation study in a disciplined fashion. In this work we propose a set of guidelines for properly capturing and representing the goals of the simulations and the metrics, allowing and evaluating the achievement of a simulation objective. We take inspiration from the goal-question-metric approach and with the aid of a specific problem formalization we are able to derive the right questions for relating simulation goals and metrics.


I. INTRODUCTION
A GENT-BASED simulation studies are effective tools for addressing problems that in real life require several resources.Despite their power, the key of success in a simulation study is to follow a comprehensive process for building simulated systems that produce acceptable and credible results.
In particular, it is widely known that a very important activity of a simulation study is to clearly establish the simulation objectives.Such objectives, or simulation goals, are the guiding element of a simulation study, thus they have to be defined at the beginning of a simulation process [1][2][3][4] [5].
In classical simulation studies, the simulation team should develop a list of specific questions that the simulation model should address and develop a list of performance metrics for evaluating goal satisfaction [1] [6].How to perform this activity is not clear enough.It is often committed to the experience of the simulation team.
We claim that problem formalization assumes a very important role in the identification of simulation goals and in this paper, we propose to identify simulation goals, starting from an ontological representation of the problem domain, and then to identify the performance metrics for evaluating goals.Our approach takes inspiration from the Goal Question Metric (GQM) approach proposed by Basili et al. [7].
GQM is an approach for expressing the goal of a software engineering study.It provides a method for defining goals, refining them into questions and finally developing a set of metrics needed to answer such questions.
For a couple of years we have been working on the development of a methodological approach covering the entire life cycle of a multi-agent simulation study taking into account each facet of the agent based modeling paradigm.Such a methodology is founded on the premise that a simulation study is conducted for testing new hypothesis on a model of a real system (i.e.: what happens if).In so doing, we are building our approach according to the phases of classical simulation studies [1][2] but providing specialized guidelines for using agent based simulation techniques.So far, we developed the initial activities of Simulation Problem Analysis phase, namely Problem Formalization [8].Problem formalization aims at formalizing simulation problems by adopting an ontological representation of the simulation problem domain.
In this paper, we employ such a representation (namely Problem Ontology) as a guide for generating questions and specifying the metrics (and also parameters) for answering those questions thus providing a structured way in order to perform a GQM-like approach in the field of simulation studies.
The Problem Ontology has to be intended as a representation of the domain on which the simulation model is being constructed.Such a domain contains, the objects involved in the problem, their behaviors, rules and organizational aspects; all these elements find a perfect mapping with the elements Basili et al. define for applying the GQM approach.
For the scope of this paper, we want to focus on the elements that can be defined Problem Ontology: • Active Entity: an active entity allows to represent concepts that are specified in the problem statement that perform dynamic actions.
• Action: an action in the problem ontology represents concepts in the problem statement describing the fact or process of doing something, typically to achieve an aim.
• Object: it represents something to which a specified action is directed.
• Predicate: It allows to express a property, a state or more generally it allows to specify a concept.The main contribution of this work is a set of guidelines for applying the GQM approach for simulation studies that use the Problem Ontology as an input.Such guidelines result in two workproducts, the Simulation Goal document and the Simulation Metrics document.
The rest of the paper is organized as follows.Section II presents an overview of the GQM approach and points out the main elements and the rationale we used for applying the GQM to our case.Section III introduces the proposed approach for simulation goal and metrics identification by describing the steps to perform for obtaining metrics and parameters.Finally, in section IV some discussions and conclusions are drawn.

II. GOAL QUESTION METRIC APPROACH
The Goal Question Metric (GQM) approach, proposed by Basili et al. [7][9][10] [11], is a paradigm that links metrics with the general goal of a process or project.The aim is to generate questions, whose answers are known, thus establishing if all the goals have been reached.Metrics are the means for answering the questions.
GQM approach relates to software metrics, indeed it is based on three main principles, classifying the entities to be examined, determining relevant measurement goals and finally determining the level of maturity of the organization.This latter principle is obviously strictly related to software production and has not been considered in this paper.The initial step of a software measurement is to identify the objects (i.e: processes, products and resources) and the attributes (i.e.: internal or external) to measure.A process is an activity related to software, products are artefacts or deliverable resulting from an activity and a resource is an entity required by a process.Attributes characterize an entity.An internal attribute refers to the entity itself and may be measured by investigating only the entity without referring to its behavior; an external attribute refers to how an entity relates to its environment.
The GQM model is based on three levels: Conceptual, Operational and Quantitative.The Conceptual level concerns the definition of Goals.In GQM, a goal is defined for an object with respect to various models of quality, from several perspectives and according to a particular environment.The Operational level concerns the identification of Questions.The questions are articulated for defining models of the object under study and for characterizing the object with respect to a selected quality issue.The Quantitative level is related to the definition of Metrics.A metric is associated with every question in order to answer it in a measurable way.
GQM is typically described as a six-step process.The first three steps concern how to use business goals for driving the identification of the right metrics.The last three steps are about gathering the measurement data and making effective use of the measurement results to drive decision making and improvements.
Hence, in order to characterize a goal in a quantifiable way, it is necessary to relate the goal to an object, or an entity, that possesses some kind of attributes related to itself or to the environment.A goal or a question may be ascribed to, at least, a couple entity -attribute.The answer to a question means that we give a measurement to an attribute.In particular, the GQM approach uses three elements for specifying the goal: purpose, issue and object.We base our work on this assumption and extend the first three steps of GQM.In particular, we provide guidelines for identifying elements the simulation goals are related to by adopting an ontological formalization of the problem domain.

III. SIMULATION GOAL AND METRICS IDENTIFICATION
All simulation studies, in general, and agent simulation study, in particular, greatly depend on how the simulation problem analysis is conducted.Simulation problem analysis aims at identifying the goals of the simulation study with respect to the domain under study.
During simulation studies some hypothesis are investigated through manipulating some independent variables that affect dependent variables and consequently the validation of the initial hypothesis.In this process, the identification of the simulation goals during the early analysis phase provides a prerequisite for identifying metrics, or parameters, for the evaluation of the simulation hypothesis.
Sentences like "Improve the timeliness of change request processing during the maintenance phase of the life cycle of a system" that can be found in the description of the simulation study, hence the in some kind of problem statement, give a hint on what the simulation study is intended for, the objective that has to be reached.Sometimes, it is not obvious or immediate to identify the independent or observable variable related to the simulation goal.
In this section we show the rationale we propose for conducting two activities: the Simulation Goal Identification and then the Entity Parameter Identification.These activities aim at identifying and modeling the goal(s) of the simulation study and the related parameters to be considered for the agent-based software development.These two activities are strictly tied and are part of a complete methodological approach for supporting agent simulation studies.We claim that the identification and the representation of simulation goals greatly descend from the problem formalization.
Problem formalization is the key element in all engineering activities for producing software products of high quality.Complete and well defined simulation models require a careful analysis of the problem domain and a detailed problem formalization phase [1] [12].
We propose to join the problem formalization to the Goal Question Metric approach.We already discussed how to perform problem formalization and the proposed results are shown in [8] [13].The result of the Problem Formalization activity is a work product, the Problem Ontology diagram, containing all the elements representing the domain under study (see section I).In what we propose, the Problem Ontology may be used as a guide for generating questions and then specifying the measures for answering questions in a GQM like approach.
Using GQM, a goal is expressed through the triplet <purpose, issue, object>; it defines an object with respect to a purpose and some quality issues.This kind of triplet, in the GQM, is the starting point for extracting all the useful parameters for a specific situation.
In a simulation problem statement the goal is generally expressed through a sentence and, from the analysis of this sentence, it is possible to identify the previous triplet for applying GQM.Suppose the Problem formalization activity 1492 PROCEEDINGS OF THE FEDCSIS.GDA ŃSK, 2016 resulted in a Problem Ontology diagram, where all the active entities, the objects, the actions and the predicates, representing the simulation study, have been represented 1 .The steps we propose are: • Identify goals -the inputs of this step is the Problem Statement.An analysis of sentences in terms of nouns and verbs is required The work to be done starts with the identification of the goal, or the goals, of the simulation study; this activity implies a careful analysis of the Problem Statement in order to identify sentences from which goals may be extracted.Since, as Basili et al. [7] highlight, "typical goals are expressed in terms of productivity, quality, risk, satisfaction", useful sentences are those containing verbs such as assess, improve, evaluate and so on.Sometimes, the sentence may be in the affirmative or in the interrogative form and some others it may not be simple to identify the sentence related to the simulation goal.In this latter case a refinement of the problem statement should be required.
• Identify <purpose, issue, object> -This step implies to analyze the syntax of the sentence containing the simulation goal.The verb is generally followed by a direct object and then by a genitive case (if not the sentence may be reformulated).For instance, the sentence "Improve the timeliness of change request processing during the maintenance phase of the life cycle of a system" leads to the triplet <improve, timeliness, change request processing>.
• Prepare questions -The main input of this step is the Problem Ontology diagram (POD).In order to obtain all the possible questions for one goal, it is required to explore all the relationships between the object in the triplet and all the other elements in the Problem Ontology diagram.If there exist a relation between an element of the POD and the object of the goal then there may be some parameters, attributes or metrics that once modified may affect the object of the goal and, at the same time, the overall simulation thus giving means for assessing or evaluating the initial hypothesis the simulation study is created for.For instance, suppose that the change request processing is related to an object like time, then it should be possible that the amount of time is a value of a parameter that may affect the improvement of the timeliness.It is worth to note that two situations may happen.The POD may present the object of the simulation model or not.The first situation happens when the object is a product or a resource and it has been identified during the previous activity as an «object» or an «active entity».The second situation, instead, happens when the goal's object is a process or simply it has not been identified during Problem Formalization, in this latter case a refinement activity of the POD is necessary and should be performed. 1The reader may have a look at [8] for a detailed example.
The Problem Ontology diagram is very important in all the previous steps.The way we employ POD is the very contribution of this paper in fact it provides an important means for supporting the analysis and identification of goals and parameters.The efficiency increasing in the identification of the right elements to consider is also obtained.
The work products resulting from the Simulation Goal Identification and the Entity Parameter Identification are Simulation Goal (SG) document that is composed of two different diagrams and the Simulation Metric (SM) document.The first contains a diagram representing the view on the simulation goal(s), its relationships with the elements of the POD and some possible new goals coming from reasoning about the relationships among the main simulation goals and POD objects.In the second the simulation goal(s) is illustrated with all the new goals identified and the kind of contribution they give to the main goal(s).
A brief example of the resulting work products is given in Fig. 1.a) and Fig. 1.b) as regard the SG and Fig. 1.c) as regard SM.These documents are related to a specific case in which, from the Problem Ontology, the analyst has identified the sentences "How may we improve the value of the throughput of the warehouse?"and "Another aim of this study is to reduce the costs of warehouse managing".The related triplets are: <improve, value, throughput> and <reduce, cost, management>.By Applying the proposed approach and by iterating them on the elements of the POD a portion of the obtained results is shown in the figures.The second work product shows questions and parameters.Obviously, for space reasons, this example cannot be complete and exhaustive but is intended to give a view on the work products.

IV. DISCUSSIONS AND CONCLUSIONS
Determining the goals of a simulation study is a very important task because from these we may obtain the set of metrics and test parameters that are typical of scientific experiment, such as simulation ones.
Simulation goals identification is generally a mental and speculative activity tied to the skills and knowledge of the analysts and very few methodological recommendations exist in literature.Since we need to extract some metrics and parameters, that are formal elements, we want to apply a disciplined way in order to avoid the risk of forgetting or omitting some important elements of the problem domain.
In this paper, we have proposed a GQM like approach for determining such a parameters starting from the simulation goal identification.This work is a further step of a more complete one: a complete design methodology for developing agent based simulation studies, this methodology also includes the agent-based system development.
One of the most important parts, already developed, is the problem domain formalization that becomes the most important and necessary input to the simulation goals identification.In some previous works we described how to represent the problem domain as an ontological (POD) representation In the present work, we combine that with the well known goal-question-metric approach (GQM) in order to create a disciplined and structured method for the identification of the goals and the related metrics and parameters useful for understanding if the goals have been satisfied for the simulation system.
The use of a problem formalization and of the POD ensures a final result coherent with the domain under study and complete since the POD, while applying GQM, is explored in its entirety.For sure this approach depends on the presence of a good and well done Problem Ontology Diagram but, the GQM approach is a valid support that, once applied, also allows to reason on the problem domain and on the POD and potentially to refine it.The same we make with the Problem Formalization [8], both are iterative and incremental approaches that lead to the goal simulation model and at the same time allow to refine the POD towards a status where, reasonably, nothing has been omitted.
In the future, we plan to complete the methodological approach and to validate the whole approach using the true experimental set up coming from the simulation model.