Sensitivity Analysis of the Criteria Weights Used in Selected MCDA Methods in the Multi-Criteria Assessment of Banking Services in Poland in 2022

The main purpose of this article is to compare the sensitivity of the weights of individual criteria in the assessment of the most popular banks' websites and the impact of the MCDA methods applied on the results of these assessments. The analysis was carried out for three selected, the most popular and different assessment methods: TOPSIS, VIKOR and PROMETHEE II. The evaluation of the websites was made on a sample of 350 bank customers, whose opinions were obtained using the CAWI method using a survey form. The survey made it possible to distinguish the 16 most popular banking services in this group, and only these banks were then evaluated. The survey questionnaire was obtained after verification of the pilot version created on the basis of previous research. The websites most known to the respondents were tested using three MDCA methods: TOPSIS, VIKOR, PROMETHEE II. The sensitivity of the results in each of the banks to the development of weights for 18 attributes (service evaluation criteria) was examined. The obtained results indicate the possibility of interchangeable use of the distinguished assessment methods, which may be helpful for business practitioners when analyzing and designing banking services.


I. INTRODUCTION
T O PREPAREhe main purpose of this article is to deter- mine the sensitivity of the position of the websites of the rated banks (C1, ..., C16) in the respondents' rankings depending on the weights of the distinguished assessment attributes (A1, ..., A18).Evaluations of websites, [1] assessed by respondents at the end of 2022, allowed for the construction of a common data set.Based on this set, three experiments were performed: • comparison of the results obtained from the simple point method with the new, original MDCA method [2], [3] the Conversion method [4], • banking services were assessed and the five most popular MDCA [5], [6], [7] methods: TOPSIS [7], COMET [8], VIKOR [9], PROSA-C and PROMETHE II [10] were compared, in terms of convergence of results, to see if the situation observed in the first experiment also occurs This work was not supported by any organization among the methods already widely known and recognized .The obtained results indicate significant deviations of the applied VIKOR [11] method from other methods.Thus, the discrepancies in the results of the Conversion method in this light did not seem to be something exceptional after this study, • the most important factor determining the assessment, and thus the position of individual banking services in the ranking, are their attributes, a priori determined at the beginning of the study.Therefore, the sensitivity of the position in the ranking of banking services depending on the change in the weights of these attributes was examined.This article presents the results of a recent experiment.It is a continuation of research undertaken on the application of multi-criteria methods [12], [13] for evaluating the latest IT solutions in the economy [14].These studies are both methodical and practical.Because, as it turned out, there is no single, universal method of assessing usability [15].The results depend primarily on the selection of parameters (attributes) -evaluating the studied phenomenon.These parameters vary depending on the studied phenomenon, industry, end user, purpose of the results, etc. [16].
This study uses a constantly modified set of criteria agreed in 2008 with the best specialists in Poland dealing with research in the banking industry.During subsequent crises, it was modified arbitrarily by the authors (external factorshigh inflation, pandemic) and by end users (bank customers) for reasons of comprehensibility and preferences.External factors extended the list of attributes, which in turn, after verification by a pilot group of users, was reduced to the most understandable attributes (after possible corrections) and important from the client's point of view [17].
The problem directly resulting from the selection of attributes are the significance weights assigned to them, which in a sense reduce the subjectivity of the final methods [18].The simplest method to solve this problem is, of course, to ask end users about the level of significance of a given criterion, and to take the average of their answers.
However, a different approach is often used that simulates the results depending on the level of weights assigned to them.Sometimes it is assigning weights to specific types of attributes (in the case of banking services: economic, technical and anticrisis), and sometimes to each assigned attribute and observing the results [19].The simplest method to solve this problem is, of course, to ask end users about the level of significance of a given criterion, and to take the average of their answers.
However, a different approach is often used that simulates the results depending on the level of weights assigned to them.Sometimes it is assigning weights to specific types of attributes (in the case of banking services: economic, technical and anticrisis), and sometimes to each assigned attribute and observing the results.

A. Research procedure
The research procedure in this case included the following steps: • bibliographic analysis of website evaluation using MCDA methods, • construction of a pilot version of the survey questionnaire, • verification of the survey form and preparation of its final version, • random selection of groups of respondents and inviting them to complete a questionnaire using the CAWI (Computer Associated Web Interview) method, • obtaining data and initial verification of their correctness, • selection and justification of methods for evaluating banking services in order to make calculations and obtain results as well as to make comparisons between them, • analysis, discussion and comparison of results • drawing conclusions, defining limitations and further directions of research.Due to the research conducted earlier and the popularity of the analyzed methods, two of them were initially selected in the first experiment: the simple point method and the Conversion method [20].Interpretation and comparison of the results obtained with these methods and their differentiation by means of the Euclidean distance were performed.Then, for five consecutive MDCA [14], [21], [22] methods: TOPSIS [23], COMET [24], VIKOR [25], PROSA-C [26] and PROMETHE II [27], calculations were made, ranking lists were prepared and the results were compared.In the analysis of the results, the calculated values of the preference function for subjective weights were taken into account.Then, the correlations between the rankings obtained with different methods and the Euclidean distances were calculated in order to examine the level of differentiation of the results obtained between the individual pairs of the methods used.The above analyzes were the content of the previous two articles.
However, this paper presents a sensitivity simulation.It consisted in the fact that for each method (TOPSIS, VIKOR, PROMETHEE II) [28], [29], [30] the values of the weights of individual evaluation criteria (A1,. . ., A18) of the existing selection variants (C1 -C16) were successively modified.The value of the weights was changed successively, and the weights of the remaining criteria were adjusted -in equal proportions, so that the weights of all the criteria add up to 1.

B. Sample characteristic
The rankings of websites were based on data collected using the CAWI method in autumn 2022.They covered 356 people, with over 48% survey response.A five-point, simplified, standardized Likert [31] scale was adopted to assess individual criteria: • 1.0 -fully meets the requirements of the criterion, • 0.75 -almost completely meets the requirements of the criterion, • 0.50 -meets the requirements of the criterion on average, • 0.25 -meets the requirements of the criterion at least, • 0.00 -does not meet the requirements of the criterion.
The original form of the questionnaire was verified on a pilot sample of 50 people, conducted in an academic environment.Individual criteria -attributes of banking services -were examined in terms of their comprehensibility and importance for an average website user.After verification, corrections and removal of the least important criteria, 18 attributes were taken into account for the assessment of each website, divided into the following three groups: economic, technological and anticrisis.A detailed list of attributes is included in Table 1.
The evaluation was conditional upon evaluating the websites of a well-known electronic bank in comparison with the website of another banking website.This condition resulted from the desire to obtain answers from experienced respondents dealing with various electronic banking services.Thus, a total of 712 full banking service ratings were received, as some respondents rated two or three banks.
Respondents rated the sixteen (A1, A2, . . ., A16) the most frequently used banking websites.They can be found in Table 2.All banking services that received less than five ratings are not included in this list -the ratings of 16 banks were rejected.
The research sample was selected in a diversified way: on purpose -the research was carried out in the academic environment in randomly selected didactic groups and using a link to the online survey [32].
The survey was mainly aimed at young people.The age range therefore ranged from 19 to 35 years).This choice could have influenced the results of the survey (41 million people in Poland are potential customers of online and mobile banking, over 54% of registered customers are active users of online banking and 44% of active users of mobile banking).The surveyed age group constitutes over 65% of users) .Among the surveyed respondents there were over 70% women and nearly 30% men.19% had bachelor's and incomplete higher education, and 80% had secondary education.The largest group of people came from large cities (over 200,000 inhabitants), and 19% from rural areas.One fourth came from small, medium and large towns -up to 200,000.inhabitants.TOYOTA Bank Polska SA Among the respondents, there were 52% of students, 31% of people working under a contract for specific work, mandate or running their own business and 17% working under an employment contract.The most frequently performed occupations are: office workers (63%), service workers (16%), specialists (8%) and workers employed for simple technical work (7%).
Data on the assessment of banking services are generally relatively homogeneous and consistent.After obtaining them, a reliability test in the form of Cronbach's alpha coefficient was applied.For all attributes, the Cronbach's alpha coefficient indicating the internal consistency and reliability of the sample [68] was greater than 0.75.The measure of internal consistency of the 16 dependent variables, based on Cronbach's alpha coefficient, was 0.85 (0.94 for Cronbach's alpha calculated on the basis of standardized items), for 18 items in total.

III. ANALYSIS OF RESULTS AND DISCUSSION
Sensitivity analysis is a technique that studies the effect of changes in one of the independent variables that make it up on the dependent variable of any model.In order to study in detail how the changes in weights affect the final ranking, a sensitivity analysis was performed.For each criterion, nine evaluations were performed, where the criterion in question was given weight 0.1, 0.2, . . ., 0.9 while all other criteria were set to 0.5.This allowed to study the effect of each particular criterion on the overall ranking.A total of 162 evaluations was therefore performed.The results of these evaluations were then plotted and are presented on figure1 and figure2.
Here are shown results of the sensitivity test to changes in the weights for individual attributes (A1, A2, . . ., A18) successively, for all sixteen analyzed banking services (C1, C2, . . ., C16).Place -position in the ranking (on the y-axis) 1 -is the best, 18 -is the worst.On the x-axis, there are weights of the selected criterion (its symbol is in the chart title).The weight values in all cases changed every 0.1.Examination of the figure below shows that although sometimes very minor,  The analysis of the sensitivity of the position in the ranking of individual banks to changes in attributes leads to the following assessments.For C1, the key attribute in the TOPSIS method is A7, which, when raised to the value of 0.3, shifts the significance of this parameter to the first position.It reacts similarly to changes in parameters A2, A3 and A8.Similar behavior can be observed in the VIKOR method, but it requires a value of 0.5.However, the ranking positions for C1 behave separately and completely chaotically.For C2the key attribute is A7, a slight increase in its value moves its position from 13th to 1st in the TOPSIS and VIKOR methods.In the PROMETHEE II method, this change, as in the case of other parameters, increased the position by only one position.For the C3 bank -the most important feature is A10, after raising the weight to the value of 0.2-0.3, it moves to the first position after using the TOPSIS and VIKOR methods, in the PROMETHEE II method, the A10 attribute remains firmly in the first place.For the A4 bank, such a key attribute is also A10, after increasing the weight of this parameter (using the TOPSIS method), it moves to the first place in the ranking, the attribute A9, which previously occupied it, moves to the thirteenth place.It is similar in the other two methods, in PROMETHEE II, it is still the most important attribute, regardless of the weight assigned to it.
The situation is slightly different for the C5 bank.Here, the most important parameter is A7, which with the increase of the weight value moves from the twelfth position first to the third, and then with the weight equal to 0.5 -to the first place.A similar "jump" can be observed in the VIKOR method, while in the PROMETHEE II method this process is more stable.For the C6 bank, the most important attribute is A10, while the most sensitive to weight change is A7, which was moved from thirteenth to third position in the TOPSIS method.Such a tendency is shown by the attributes A10 and A7 in the VIKOR method, while in the PROMETHEE II method they always stop at the first and third position, regardless of the values of the adopted weights.A similar situation occurs for the C7 bank, where A7 moves from the twelfth to the first position in the ranking, with weight=0.9,dealing with the A10 parameter (TOPSIS method).When using the VIKOR method, this scheme is almost duplicated, the application of the PROMETHEE II method shows a low sensitivity of these attributes to the change of weights and leaves them respectively on A7 in the first position, A10 in the second position.For banks C8-C14, the situation almost repeats itself, as the weight increases, the attribute A7 from the further position is moved to the first position, replacing the parameter A10 (except for C9, where A10 remains in the first position all the time as the weight increases).For all the cases mentioned so far, the attributes A1-A4 and A8 are relatively stable and independent of the weights and the assessment method adopted, sometimes only changing places in the ranking.That is, for these banks the most important attributes are: security and the amount of the bank card fee (if any), and the least important: nominal annual interest rate on personal accounts, keeping an account in PLN/month, fee 1220 PROCEEDINGS OF THE FEDCSIS.WARSAW, POLAND, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
for transfer to the parent bank, and fee for transfer to another bank and additional services.The situation is completely different for the C15 bank.In the TOPSIS method, A8 moves from position 15 to the first position and A2 from the twelfth position to the second position.As the weights increase, parameters A9 and A5 are lost.Finally placing respectively in: fifth and sixth position.This phenomenon is repeated for the VIKOR and PROMETHEE II methods, where there is also the strongest so far "shaking" of the attributes' positions due to the height of the simulated weights (all of them change their place in the ranking).The sensitivity of the attributes to the change of weights for the C16 bank is also different.The first two positions A8 and A9 are relatively stable, regardless of the calculation method.In the TOPSIS and VIKOR methods, the remaining attributes undergo significant changes in the ranking position.In the PROMETHEE II method, the remaining attributes undergo far-reaching changes in position.An interesting phenomenon is the fact that the parameters A7 and A10 for the last two analyzed cases are moved to the last positions in the ranking as the weights increase.The analysis shows that for the customers of these two banks the most important features may be: additional services and account access channels, and much less important attributes related to fees.Technical parameters turned out to be the least sensitive to weight changes in all cases.

IV. CONCLUSIONS This work was aimed at comparing:
• sensitivity of the results of the multi-criteria evaluation of websites of the most popular banks in Poland to changes in the weights of the attributes used for this evaluation, • the results of changes in attribute weights in three selected MDCA methods: TOPSIS, VIKOR and PROMETHEE II.The results obtained as a result of three experiments allow us to conclude that for the proper evaluation of a multi-criteria problem, certain conditions must be met: • firstly, it is necessary to select the evaluation criteria (attributes) characterizing the most important features of the analyzed issue from the user's point of view, • secondly -the selection of a method that will guarantee that the collected data will be properly used, • thirdly -the method of comparing the results obtained, • fourthly -it is also recommended to select the appropriate criteria weights, the structure of which may reflect the preferences of the decision maker or the client.The current research shows that the most comparable rankings were obtained using the TOPSIS and COMET methods, while the greatest differences in relation to the results obtained from other methods were observed using the VIKOR method.On the other hand, the results obtained using the VIKOR method were closest to the results obtained using the TOPSIS method.
When testing the sensitivity of the results to changes in the criteria weights during the experiments, the results obtained with the TOPSIS and VIKOR methods behaved similarly.Separate results were obtained for the results obtained using the PROMETHEE II method.
It seems that it is difficult to judge the optimality of the methods used or their objectivity even after this series of experiments.However, it cannot be ruled out that the combination of several methods, e.g.simple methods for preliminary analyzes and complex methods such as VIKOR or PROMEYHEE II for more comprehensive results, would give positive results.
The main barriers of this work were the limitation of obtaining data to academic representatives of generation Z, which on the one hand indicates the main, future users of banking services, but on the other hand makes it difficult to generalize conclusions and use a limited number of the five most popular MCDA complex methods.
Nevertheless, the results of the conducted experiments encourage to continue research in order to expand the set of MCDA methods enabling the pursuit of convergence of results and thus the objectification of assessments in the analyzed area.Also, due to the importance of the banking sphere and the importance of the tool of communication with the user, which is the website, this direction will dominate in future research.

Fig. 1 .
Fig. 1.Sensitivity to simulated changes in the weights of individual attributes in the analyzed banks, banks C1-C9; Source: own study

Fig. 2 .
Fig. 2. Sensitivity to simulated changes in the weights of individual attributes in the analyzed banks, banks C10-C16; Source: own study

TABLE I LIST
OF ATTRIBUTES OF BANKING WEBSITES ASSESSMENT; SOURCE: OWN STUDY

TABLE II LIST
OF ATTRIBUTES OF BANKING WEBSITES ASSESSMENT; SOURCE: OWN STUDY