SciELO - Scientific Electronic Library Online

 
vol.18 número84La interculturalidad: una mirada desde el enfoque cognitivo, comunicativo y socioculturalGestión de la socialización organizacional de los directores escolares noveles: fundamentos conceptuales, orientadores y referenciales índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

  • No hay articulos citadosCitado por SciELO

Links relacionados

  • No hay articulos similaresSimilares en SciELO

Compartir


Conrado

versión impresa ISSN 2519-7320versión On-line ISSN 1990-8644

Conrado vol.18 no.84 Cienfuegos ene.-feb. 2022  Epub 10-Feb-2022

 

Artículo Original

Automation of the performance indicators assessment of scientific and pedagogical workers: structural analysis

Automatización de la evaluación de los indicadores de desempeño de trabajadores de la ciencia y pedagogos: un análisis estructural

Maxim Sergeevich Logachev1  * 
http://orcid.org/0000-0002-0425-5014

Irina Viktorovna Kulibaba1 
http://orcid.org/0000-0002-8700-8763

Sergey Vladimirovich Nazarenko2 
http://orcid.org/0000-0001-6120-163X

Tatiana Nikolaevna Seregina2 
http://orcid.org/0000-0001-7534-3698

Maxim Romanovich Logunov3 
http://orcid.org/0000-0001-7582-837X

1 Moscow Polytech University. Russian Federation

2 Financial University under the Government of the Russian Federation. Russian Federation

3Higher School of Economics. Moscow. Russian Federation

ABSTRACT

The article presents a study of the process of assessing the effectiveness of scientific and pedagogical workers in the educational organization of higher education. The result of such an assessment is the basis for determining the activity of teachers for a certain period and the accrual of a stimulating part of the salary. The purpose of the study is to automate the methodology for evaluating teacher performance indicators by implementing an information system. The main task of such a system should be the ability to reallocate the resources required to implement the assessment process in such a way as to reduce the burden on individual specialists and reduce the number of manual checks of individual criteria. The paper defines the features of the teacher evaluation process, the subjects, and objects involved in its implementation, as well as the mechanisms of their interaction. Based on the results obtained, models have been developed, problems have been identified, and ways to solve them have been determined. The paper presents models of processes that allow evaluating changes after the introduction of some modules of the information system.

Keywords: Education; education management; educational organization; employee efficiency; quality system

RESUMEN

El artículo presenta un estudio del proceso de evaluación de la efectividad de los trabajadores científicos y pedagógicos en la organización educativa de la educación superior. El resultado de tal evaluación es la base para determinar la actividad de los maestros durante un período determinado y la acumulación de una parte estimulante del salario. El propósito del estudio es automatizar la metodología para evaluar los indicadores de desempeño docente mediante la implementación de un sistema de información. La tarea principal de un sistema de este tipo debería ser la capacidad de reasignar los recursos necesarios para implementar el proceso de evaluación de tal manera que se reduzca la carga de los especialistas individuales y se reduzca el número de verificaciones manuales de los criterios individuales. El trabajo define las características del proceso de evaluación docente, los sujetos y objetos involucrados en su implementación, así como los mecanismos de su interacción. A partir de los resultados obtenidos se han desarrollado modelos, se han identificado problemas y se han determinado formas de resolverlos. El trabajo presenta modelos de procesos que permiten evaluar cambios luego de la introducción de algunos módulos del sistema de información.

Palabras clave: Educación; gestión de la educación; organización educativa; eficiencia de los empleados; sistema de calidad

Introduction

Modern higher education in Russia should meet international requirements and be distinguished by a qualitative approach to the learning process (Nazarenko, 2018; Borovkova, 2019). As an indicator of the quality of education, the level of competence and interest of academic staff (hereinafter referred to as ACS) can be considered. One of the approaches to evaluating this indicator is a rating system.

The use of a rating assessment of the work of ACS of an educational organization of higher education allows for a mathematically based analysis of its effectiveness for different reporting periods and is the basis for the development of a system of management actions to improve the effectiveness of its work (Borovkova, 2019).

The integral characteristic of such an assessment allows specifying the rating of objects in their ordered sequence to support the decision-making process (Logachev, et al., 2021).

Assessment of the quality of work is part of the model of state and public accreditation of Russian educational organizations. This assessment has many aspects:

  • Qualification of ACS,

  • participation in the training of ACS,

  • development of educational and methodological and material support,

  • implementation of research work,

  • participation in the work of an educational organization,

  • participation and preparation of winners or prize-winners of professional competitions, etc. (Borisova, 2016; Bortnikov & Logachev, 2020).

Thus, the work of the ACS is related to a special type of activity, where various options for evaluating its effectiveness, efficiency, and quality are possible. This circumstance leads to the presence of various approaches to the construction of evaluation models. Such models differ significantly from each other both in terms of methodological grounds and in terms of practical implementation procedures (Irzaev& Muradov, 2014).

The evaluation of the ACS performance is important not only for the quality of education but also for determining the incentive part of wages, which is an integral part of the wage system (Belov, 2019; Bortnikov & Logachev, 2020). There has been a system in all educational organizations for a long time, for determining the incentive part of the salary based on determining the rating of each teacher based on the method of multidimensional analysis (Bortnikov & Logachev, 2020).

This methodology is being improved by redefining the evaluation criteria, evaluation methods, etc. The main problem is the lack of software tools that allow performing an assessment using a minimum amount of resources without reducing the quality of the results obtained (Artemenko, 2015). Digitalization of the processes of control, monitoring, accounting, and analysis of ACS activities makes it possible to combine additional data with rating assessments, which allows obtaining detailed management decisions that are adequate to practical needs (Ryabova, 2011).

Materials and methods

The method of structural analysis was used to conduct a study of the subject area. The structural analysis allowed transforming inaccurate knowledge about the original problem domain into accurate models that describe the subsystems of the modeled processes or objects. This allows accurately determining:

  • all actors involved in the process of assessing the effectiveness of ACS.

  • Key objects of both the processes themselves and external objects that affect the processes of the subject area.

  • Relationships between all objects and processes.

  • Methods and means of information system modeling.

Visualization of the results of the structural analysis allows specialists to identify all the ways to achieve goals (including alternative ones), establish dependencies between all objects and processes, and identify "bottlenecks" in the implementation of each of the processes (Logachev 2020ab). In this case, the "bottlenecks" are those objects or time gaps in the implementation of processes that can potentially increase the time spent working with data, their distortion, or loss. Also, the use of such models allows:

  • Determining the range of employees of the organization who can or should perform functions for the implementation of the relevant processes of the subject area.

  • Setting the conditions (both standard and with certain deviations) for performing functions.

  • Determining the resources needed to implement the processes.

  • Developing options for changing the implementation of the process to increase its efficiency in certain parameters (time, resource use, etc.).

  • Identifying risks in the implementation of processes (Logachev, 2020b).

The obtained results of the structural analysis are presented in the form of the following graphical models:

  1. The BPMN model. It is possible to present an algorithm for executing a process with a display of all possible conditions and events that may occur during the entire time of its implementation. Therewith, this model clearly defines all the participants responsible for performing individual tasks that are part of the process (Logachev, 2020a).

  2. Diagram of the internal structure of the information system. Allows visually showing the internal structure of an information system, i.e. all its modules and other objects (for example, a database), as well as the relationship between them.

  3. EPC diagram. It is used to visualize the chain of events and actions, their participants, and documents (Logachev, 2020b). The use of such a model in the study allows determining the interaction of specialists of an educational organization with the modules of the information system being developed at different stages of the implementation of the process of evaluating the effectiveness of the ACS.

Results and discussion

The organization of the process of assessing the quality of the ACS work in educational organizations was carried out by collecting documents confirming the presence of achievements, and a completed assessment sheet. In the evaluation sheet, each employee performs a self-assessment for each criterion. Then, the self-assessment is checked by the employees of the educational organization responsible for the relevant areas of activity (Vavilova & Chernousov, 2019; Galiullina, 2019). This process is automated in some educational organizations. Automation, in this case, consists of the fact that the evaluation sheet becomes electronic and the employee has the opportunity to download electronic versions of supporting documents (Logachev & Samarin, 2016). In general, this process can be represented as a BPMN model in Figure 1.

Source: Bortnikov &, Logachev (2020).

Fig. 1 - BPMN-diagram of the ACS rating process of an educational organization. 

The result of the evaluation process is a certain amount of points that characterize the activity of the ACS for a given period.

The criteria of the evaluation list include the activity of the teacher both within the educational organization (participation in internal events) and outside it (publication activity, preparation of participants for professional skills competitions, broadcasting of pedagogical experience, reports on scientific activities, etc.) (Ivanov, 2015; Grigorash, 2019). The documents confirming the activity of the teacher include:

  • Orders of the heads of the educational organization.

  • Certificates (diplomas, certificates).

  • Articles (or any other publications in periodical and/or non-periodical publications) published in citation databases (for example, RSCI, WebofScience, Scopus, RussianScienceCitationIndex, etc.) (Bortnikov & Logachev, 2020).

These documents are collected by the responsible employee of the organization (for example, the secretary of the structural unit) for further transmission to the quality service. If there is no quality service, then the evaluation process is supported by the staff of the methodological service (or any other service that is assigned such labor functions). Upon completion of the audit, the evaluation sheets are submitted for consideration by the commission, which shall approve the results obtained (Ryabova, 2011; Stain, 2018). Based on the results obtained, an order is formed for the payment of incentive funds to the ACS.

Thus, the following disadvantages can be distinguished in the system of organizing monitoring of the ACS effectiveness:

  1. "Non-transparent" assessment (lack of assessment control, the biased attitude of the process subjects to the teacher being evaluated, etc.).

  2. Problems of organizing the processing, storage, and transmission of information presented on paper.

  3. Inefficient use of resources

    • Material (duplication of documents on paper)

    • Temporary (increasing the time for processing evaluation sheets, creating reports, and maintaining statistics).

    • Labor (verification of criteria by visual reconciliation and digitization of results).

The use of the information system for the implementation of the process of evaluating the activities of the ACS should provide the following features:

  1. Storing accounting records. The account of the creator of the assessment sheet criteria, teachers, and ACS is formed by integrating data from the existing system in the educational organization (for example, IDM, IAM, 1C, etc.). This creates a unified system of software products used in one educational organization and a consistent database of various documents (Logachev, et al., 2019).

  2. Intuitive and responsive interface. This ensures a convenient and fast user interaction with the system's functionality. Also, the user should be able to customize the interface to suit their characteristics (for example, choose the color scheme, font size, etc.).

  3. Customizable evaluation sequence. Each criterion is assigned an appropriate specialist who performs the verification. Therewith, groups of verifiers can be assigned for individual criteria. Each reviewer can only see their part of the criteria and the points awarded. This eliminates the possibility of not only a biased attitude of the examiner but also a deliberate reduction in the number of points, for example, due to a high current result (Bortnikov & Logachev, 2020).

  4. Automatic decision-making. The system should be able to automatically add data from different sources (for example, from citation databases, reports, downloading data from systems used in an educational organization, etc.). This can be done by keywords, by executors of orders, etc. This approach would allow each teacher to confirm the matching of documents for their subsequent participation in the evaluation system.

  5. Uploading reports based on the results of the audit. When exporting the results, it is possible to configure the parameters. This allows keeping statistics, preparing reports for creating internal local reports, etc.

The modular structure of the information system. It is a classic client-server application with the ability to use external API services (Bortnikov & Logachev, 2020; Chernyshov, 2020). Thus, the structure of the information system can be represented using the diagram in Figure 2

Fig. 2 - Structure of the information system for monitoring the effectiveness of the ACS. 

The frontend includes:

  • Core-functionality (filling out the evaluation sheet, evaluating criteria, exporting reports, etc.).

  • A library of criteria (templates for evaluation sheet components).

The backend of the system implements algorithms for the correct operation of the information system. The API part allows for the operation of a standard test library (Bortnikov & Logachev, 2020).

Figure 3 shows an EPC diagram showing the main aspects of the implementation of the process of creating a score sheet in the information system for monitoring the effectiveness of the ACS. The EPC diagram allows comparing the funds used for the implementation of the task and the specialists whose participation is necessary for its support (Grigorash, 2019).

Fig. 3 - EPC-diagram of the evaluation sheet creation process. 

The following specialists are required to implement the assessment sheet creation process:

  • secretary of the commission;

  • system administrator.

Interaction with the information system for monitoring the effectiveness of the ACS can be presented as shown in Figure 4.

Fig. 4 - BPMN-diagram of interaction between the subjects of the performance monitoring process using the information system tools. 

The secretary of the commission does not take part in setting up the system and his/her functions will have to be performed by the administrator. To do this, the administrator shall have all the criteria and evaluation set approved by the commission.

The success of a software product depends on how well it meets the needs of end-users as a tool for solving various tasks (Murtazina, 2019). Requirements engineering plays a key role in managing software development projects. Requirements engineering covers the extraction, analysis, specification, and validation of requirements. In most cases, the formalization of the process of working with requirements is random (Murtazina, 2019). The development of requirements for a software product that implements the evaluation of the effectiveness of the ACS is complex. This is because different educational organizations treat this process differently: different periods or quality criteria are set, the evaluation principles do not coincide, etc. The study reflects only the main processes, its objects, and subjects, which are typical for most educational organizations.

The use of an information system for the process of assessing the effectiveness of ACS solves most of the problems, but the following problems may arise (Bortnikov & Logachev, 2020):

  1. Setting up individual modules of the system (for example, automatic determination of indicators) is a time-consuming process that requires a certain qualification of the specialist who performs the administration.

  2. The low level of computer literacy is common among the heads of structural divisions due to their age. This may lead to a reluctance to implement or fully use software products.

Conclusions

The human resource potential of an educational organization includes training for the performance of its functions (Nazarenko, 2015). Also, the human resource potential should include a set of capabilities of the ACS in the long term, taking into account the scientific and pedagogical qualifications, the level of motivation, practical experience, business activity, performance, and innovation of the work. All this determines the remuneration of the ACS.

The modern system of remuneration for ACS involves the creation of a rating for each teacher. The rating includes a single unified methodology with multivariate analysis. Such techniques are used in all educational organizations. Automation of the process of rating formation and monitoring according to certain indicators is obvious and indisputable. Its implementation requires a study that would show all aspects of the implementation of this process.

As a result of the work, models of the process of evaluating the effectiveness of the ACS were developed: the subjects of the process were identified, the links between them were determined, as well as the problems that may arise during its implementation.

As a solution to this problem, a model of an information system was proposed that implements the process of evaluating the activities of the ACS. The presented model of the system evaluates the activities of the ACS based on various criteria based on the requirements of both the educational organization itself and external agencies. The system assumes a combination of manual data processing and automatic verification mode. The automatic verification mode is implemented by integrating additional modules that interact with other software products used both in the educational organization itself and outside of it.

The developed model of the information system is flexible, quickly configurable, scalable, and provides a minimum of duplication of functionality with other software products used in the educational organization.

Bibliographic references

Artemenko, O.V. (2015). Avtomatizirovannaya sistema otsenki effektivnosti truda professorsko-prepodavatelskogo sostava po publikatsionnoi aktivnosti. http://web.snauka.ru/issues/2015/01/43373Links ]

Belov, M.V. (2019). Modeli i metody soglasovannogo upravleniya zhiznennymi tsiklami organizatsionno-tekhnicheskikh sistem. (Doctoral Thesis). Moscow. [ Links ]

Borisova, E. V. (2016). Pedagogicheskii kadrovyi potentsial v aspekte innovatsionnoi kompetentnosti. Vestnik Tverskogo gosudarstvennogo universiteta, 1, 90-96. [ Links ]

Borovkova, G.S. (2019). Metody i algoritmy upravleniya obrazovatelnym uchrezhdeniem vysshego obrazovaniya s pomoshchyu reitingovoi sistemy. (Doctoral Thesis). Lipetsk. [ Links ]

Bortnikov, M.E., & Logachev, M.S. (2020). Avtomatizatsiya otsenki pokazatelei raboty professorsko-prepodavatelskogo sostava. Standarty i monitoring v obrazovanii, 8(5), 26-33. [ Links ]

Chernyshov, B.A. (2020). Tsentralizovannoe upravlenie vzaimodeistviem obektov organizatsionnoi sistemy na osnove optimizirovannykh modelei reitingovogo otsenivaniya. (Doctoral Thesis). Voronezh. [ Links ]

Galiullina, A.F. (2019). Modeli i algoritmy intellektualnoi podderzhki prinyatiya reshenii po upravleniyu kachestvom predostavleniya gosudarstvennykh i munitsipalnykh uslug. (Doctoral Thesis). Ufa. [ Links ]

Grigorash, O.V. (2019). Otsenka effektivnosti deyatelnosti kafedry v sisteme professionalnogo obrazovaniya studentov tekhnicheskikh spetsialnostei. (Doctoral Thesis). Krasnodar. [ Links ]

Irzaev, G. K., & Muradov, M.M. (2014). Avtomatizirovannaya sistema monitoringa i otsenki publikatsionnoi aktivnosti prepodavatelya kafedry vuza [An automated system for monitoring and evaluating the publication activity of the teacher of the department of the university. http://web.snauka.ru/issues/2014/08/36567Links ]

Ivanov, V.A. (2015). Problemy otsenki nauchnoi deyatelnosti professorsko- prepodavatelskogo sostava i nauchnykh rabotnikov vuza. Nauchnye i obrazovatelnye problemy grazhdanskoi zashchity, 3, 100-104. [ Links ]

Logachev, M.S. (2020a). Informatsionnye sistemy i programmirovanie: Administrator baz dannykh. Infra-M. [ Links ]

Logachev, M., Orekhovskaya, N., Seregina, T., Shishov, S., & Volvak, S. (2021). Information System for Monitoring and Managing the Quality of Educational Programs. Journal of Open Innovation: Technology, Market, and Complexity, 7(1). [ Links ]

Logachev, M., Tkacheva, G., & Samarin, Y. (2019). Educational Program as a Tool of the Quality Management System of Vocational Education. INFRA-M. [ Links ]

Logachev, M.S. (2020b). Informatsionnye sistemy i programmirovanie: Spetsialist po informatsionnym sistemam. (Final qualifying work). Infra-M. [ Links ]

Logachev, M.S., & Samarin Yu, N. (2016). Razrabotka kriteriev dlya avtomatizirovannogo monitoringa kachestva soderzhaniya normativnykh dokumentov obrazovatelnogo protsessa. In, Avtomatizirovannye sistemy upravleniya kachestvom obrazovatelnogo protsessa. (Monografiya). Moscow State University of Printing Arts. [ Links ]

Murtazina, M.Sh. (2019). Intellektualnaya podderzhka prinyatiya reshenii v oblasti inzhenerii trebovanii na osnove ontologicheskikh modelei predstavleniya znanii. (Doctoral Thesis). Novosibirsk. [ Links ]

Nazarenko, S.V. (2015). Sotsiologicheskii podkhod k organizatsii sotsialnogo kontrolya v sfere obrazovaniya i nauki. Ekonomicheskie i gumanitarnye issledovaniya regionov, 3, 78-85. [ Links ]

Nazarenko, S. V. (2018). Tekhnologiya sotsialnogo kontrolya v universitetskikh obrazovatelnykh praktikakh. Pravo i obrazovanie, 7, 73-88. [ Links ]

Ryabova, T.M. (2011). Otsenka professionalnoi deyatelnosti professorsko-prepodavatelskogo sostava vuzov v usloviyakh modernizatsii vysshego obrazovaniya. (Doctoral Thesis). Moscow. [ Links ]

Stain, D.A. (2018). Kvalifikatsionno-orientirovannaya ekspertnaya sistema upravleniya obrazovatelnym protsessom vuza. (Doctoral Thesis). Chelyabinsk. [ Links ]

Vavilova, E.V., & Chernousov, A.A. (2019). Opyt razrabotki informatsionnoi sistemy reitingovaniya nauchno-pedagogicheskikh rabotnikov. Otkrytoe obrazovanie, 23, 76-85. [ Links ]

Received: November 05, 2021; Accepted: December 16, 2021

*Autor para correspondencia. E-mail: logachevmaxim@gmail.com

Los autores declaran que esta investigación no presenta conflicto de intereses.

Los autores participaron en la redacción del trabajo y análisis de los documentos.

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License