(pdf)
SciELO 
SciELO 
Permalink
Introduction
Organisation of educational process is an important process requiring a significant number of resources for its management. The ecosystem of any educational organization involves many specialists performing different functions, from teaching to preparation of documents for accreditation. All of them use software tools that can speed up the time needed to complete some job function, reduce the number of used resources, synchronize the document flow or bring information to its recipient. The development and realization of new software tools is determined by educational institutions’ need to develop and digitalize their internal processes in response to the demand for industry training specialists within the goals of the government policy.
In Russia, state program “Accessible Environment” has been developed, which aims to create due conditions and adapt the rules of social, informational and other state services for people with disabilities. This program asserts the necessity of providing information and computer-specific accessibility of the environment, introduction of new ways of interaction and promotion of goods and services that use special management tools. Educational activities should meet the requirements of accessible environment.
All this accounts for the growing research in the sphere of information and communication technologies, services for people with disabilities, the demand for specialists able to implement educational and other governmental programs aimed at providing socialization of people with disabilities, with the possibility to train appropriate specialists. Consequently, it is necessary to adapt the modern system of professional education towards realization of this area. Interpreters of Russian sign language (RSL) are professionals of this level, among the other specialists. It should be noted that RSL has become an official language of communication for hard-of-hearing people or those suffering speech impairment.
Russian linguistic universities are currently implementing educational programs to train RSL interpreters (Laamarti, 2021; Smirnova & Laamarti, 2021; Logachev et al., 2022; Smorchkova, 2022). This language is a complex system comprising hand gestures combined with facial expression, lip movements and body posture (Mohammed et al., 2019). Its separate component is the finger alphabet called dactylology, in which any letter of the national alphabet is represented by a finger symbol.
At the moment, there are numerous methodologies, technologies and software products that provide for the concept of ambient intelligent space with user behaviour analysis based on non-contact sensors, or information and reference services (Sekli & De La Vega, 2021). All this diversity provides for the solution of individual, highly specialized tasks. At times, resulting software tools exist independently and are not integrated into the overall ecosystem of the organization.
The implemented research has integrated the state-of-the-art technologies to provide the educational activities for training RSL interpreters using information system tools and additionally, using the developed software tool, to provide communication for people with disabilities in the educational process.
The development and realization of the new software tool into the ecosystem of an educational organization - the information system for monitoring Russian sign language teaching (ISCT RSL) - sets the realization of the educational process in a new way, which is reflected in the results of the present research.
Let us note the theoretical significance of the research - formalization of RSL learning control processes. The results obtained in the course of the study represent a formal process and data models creating a theoretical basis for other research in the sphere of quality management of educational process, its digitalization and possibility of operational decision-making for all subjects of the relevant business model.
The practical significance lies in the unification of RSL learning control process, systematization of accumulated experience in the relevant subject area, reduction of labour costs of routine operations by profile specialists, redistribution of their workload, as well as mitigation of subjectivity in decision-making regarding assessment of learning outcomes quality.
The hypothesis of the research is that RSL teaching control processes can be represented unambiguously and accurately as formal models that allow to identify the “problem areas” with the purpose of subsequent digitalization and reducing the share of resources required for their implementation.
The key notion in the organisation of educational process supposing the use or study of the Russian sign language is “gesture”. Realising and analysing its features, it is possible to organise communication processes between all participants of the educational process, to design, develop and implement software supporting and used in such processes, as well as to adjust the already applied information systems in the ecosystem of the educational organisation. The analysis of scholarly literature showed that a gesture is defined as some action or movement of the human body (its part) expressing some specific meaning or sense (Boulares & Barnawi, 2022). It represents a sign or symbol in its nature. The widely used gestures most often involve pointing to some object, or synchronous use of hands and body with the rhythm of the speech, thereby emphasising the words or phrases.
The scholars in their research address the historical, cultural, sociological, anthropological, psychological, psycholinguistic, artistic, philosophical, semiotic and pragmatic aspects of the gesture, as well as its comparison with the verbal language, or the study of semantics of specific gesture acts (Maltseva et al., 2021). The researchers distinguish the gesture as an independent mode of communication not reducible to the function of only complementing or explaining words.
The analysis of scholarly works in humanities showed that seemingly similar gestures may have a different meaning in different countries. The same word can be represented differently within the same language. Table 1 gives an illustrative example of difference in gestures defining a dog in the Russian and English (USA) sign languages. It should be noted that one keyframe from interpreters’ movements was selected for the demonstration (using images from www.spreadthesign.com - non-profit association European Sign Language Centre).
Table 1 - The word “Dog” in sign language.
| Language | Gesture | Note |
|---|---|---|
| English (USA), first dialect |
|
The arm is in a bent position, with movement of several fingers |
| English (United States), second dialect |
|
Consists of two gestures: touching the right thigh with the right hand and bending the arm with the clenched fist |
| Russian, first dialect |
|
The right hand at the level of the mouth makes several movements with the fingers |
| Russian, second dialect |
|
Both hands at the chest level simultaneously make finger movements |
The results of gesture demonstration of the same word from Table 1 show how important the content of the gesture is in interpreting it and assessing the appropriateness of its demonstration in teaching by relevant professionals.
The sign language enables communication for people with disabilities (Prikhodko et al., 2020). The researchers in this field highlight visual, verbal and tactile communication processes used by people with disabilities, as objects of their research. The main problem of such research is information support of such communications. Logachev in his work notes the paucity of basic research in this area in Russia and almost complete lack of means facilitating barrier-free transfer of information (Logachev & Chernova, 2022a). The main tasks of the ongoing research aim to analyse the information transformation methods, to develop the concepts of information support for relevant communications, to create due models, methods for transformation and visualisation of information as well as the algorithms for information support of communication and their software realisation for every category of people, depending on their health status (Kagirov et al., 2020; Logachev & Chernova, 2022b). In terms of the research carried out by our team of authors, the results derived by the scholars in the field of communication support for people with disabilities have theoretical and practical relevance. Thus, the currently available results make it possible to visualise information accurately in terms of static and dynamic gestures, the position of one’s hand, changes in the contour of a person’s lips. They also make it possible to see the “bottlenecks” of the educational process in organising communications for people with disabilities and in setup of software and hardware to ensure support of such communications.
A special area of sign language research should be highlighted - use of gestures as a way of interacting with the computer system. In our opinion, such research is of practical and theoretical importance, since it develops the algorithms for conversion and interpretation of gesture information and creates due software based on resulting algorithms.
The scientific works mainly address the development of software tools connected with the use of a large number of sensors, joysticks, trackballs or touchscreens, as well as complex control systems (Ryumin, 2020; The & Yu, 2021).
The ongoing research is typically based on specific features of fixation and recognition of combinations of human movements or activity of human organs (Elakkiya et al., 2021). The results of such studies are represented by models and natural interfaces (based on actual use of neural impulses, speech recognition, lip movements, facial expressions or gaze movements, fixing movements of the human body as a whole or its individual organs, etc.) (Katılmış & Karakuzu, 2021).
It should be noted that the practical and theoretical significance of such research lies in the development of methods for video analysis of human organs movement (e.g., hand), creation of algorithms and their practical realisation for automated recognition systems and organisation of various user interfaces (Kharlamenkov, 2021).
It should be noted that the ongoing research is in line with the state policy concepts for research and development in the field of information technologies, man-machine interaction, machine learning, robotics, as well as the development towards new human-machine interfaces, methods of using gestures and vision to control computer and robotic systems, infrastructure solutions and software for augmented reality.
Specialists have been globally developing, over the years, both hardware and software tools for transliteration of signs. Below are a few examples:
The University of Washington has created technological gloves (SignAloud) that transliterate the American Sign Language into English. The gloves are equipped with sensors that track hand movements and send the data via Bluetooth for analysis by due software, to be converted into the English text. After the conversion, the electronic voice articulates the resulting text. The disadvantages of this innovation include the impossibility to recognise verbal messages with further interpretation for a deaf and dumb person, as well as to transliterate the written text and take into account the facial expression or any other non-verbal elements of communication.
The specialists from Peking Union University teaching deaf people, along with the scientists from Chinese Academy of Sciences and Microsoft Research Asian team are working over creation of “Kinect” sign language interpreter. It comprises the translation and communication mode. The translation mode makes it possible to interpret sign language characters into words and vice versa. The communication mode translates sentences using 3D models. To ensure proper operation of the system, machine learning-, pattern recognition- and computer vision technologies are used (Wang et al., 2020). This makes it possible to trace the position of the sign-language communicator’s hands as well as their movement trajectory. At the same time, mutual communication is possible, since speech recognition technology provides for translation of spoken language into sign language, while the 3D model visualises messages for deaf and mute people. It should be noted that this development is experimental.
SignAll’ automated interpretation system has been developed to provide continuous communication with deaf and hearing-impaired people using American Sign Language. The system uses Microsoft’s ‘Kinect’ motion sensors and webcams with depth sensors connected. The signs and hand movements are recognised using computer vision technology, and the natural language processing system converts the derived data into a simple phrase. The technology has a potential to incorporate all of the five parameters of American Sign Language into communication for accurate interpretation of the speaker. The development has been approved by many companies (e.g. Deloitte, LT-innovate) and is supported by a Hungarian company Renawal.
The results obtained by the national and foreign scientists and developers are of particular value for the given research, since they allow the following:
to obtain models of Russian sign language objects after automated transformation, for assessment of reproduced gesture quality or the content of disabled person’s response;
to establish relationship between input and output data involved before, during and after the application of sign language transformation techniques;
to establish common access points for integration of individual modules or whole derived systems into the educational organization’s ecosystem with a view to train highly qualified professionals in the area of Russian sign language or persons with disabilities.
Me
Materials and methods
The use of any information system within the ecosystem of a company implies coordinated interaction between the objects and processes.
The characteristics and attributes made it possible to classify gestures for further software processing in order to ensure due storage in ISCT RSL database, quick retrieval and convenient use according to particular user’s objectives. Storing a reference graphic representation of a gesture (Figure 1) provides for automatic evaluation of the user’s gesture.
Fig. 1 - Keyframes of a reference gesture. (Electronic reference analytical system “Explanatory Lexicographic Dictionary of Russian Sign Language”, slovar.surdocentr.ru)
The actors involved in the educational process involving ISCT RSL include:
Students: mastering the RSL interpretation programme, people with a disability, mastering any educational programme.
Faculty members realising some educational programmes, including in the RSL domain.
Methodology department specialists providing support for realisation of educational programmes.
Experts - specialists providing qualified opinion on realisation or content of educational programmes, qualitative assessment of learning outcomes.
IT specialists providing uninterrupted operation of information systems of educational organisation.
Heads of structural subdivisions of educational organisation in charge of realisation of the educational process (e.g., classroom scheduling dispatch service, postgraduate department, etc.).
To create any information system, methodologies are used that represent interrelated processes of designing and transforming a number of consistent models at all stages of the life cycle. In order to identify the features of the processes involving ISCT RSL, the following models were to be developed:
Models defining the content of the problem domain, intended to support the design phase of a relevant software product;
Models defining specifics of realisation and customisation of existing software tools to be implemented in the educational organisation’s ecosystem;
Models demonstrating the organisation of educational process participants’ interaction during the use of a particular software product.
The development of such models was made with the use of the ‘drawio’ software tool. The authors’ choice of the software was based on a simple and intuitive interface that could be used to build graphic models of various complexity; advanced graphic import and export capabilities; possibility of simultaneous editing by several authors (synchronisation of the authors’ team’s work during each member’s remote work); support of a large number of graphic notations. Based on the above, the authors derived accurate graphic models demonstrating the objects’ actions during the realisation of any process, as well as providing for due development stage of the information system through reliable and transparent data.
The authors used structural/functional modelling methods to create their models. These methods make it possible to describe all business processes of any complexity by means of various graphical notations supported by ‘drawio’. The analysis of processes involved in control of teaching RSL or any other training programme in the educational organisation was performed by the authors using stepwise refinement method (Logachev et al., 2021). Using it, we were able to decompose the processes into separate parts in order to visualise all the links between the involved objects and subjects (Davlatov, 2021). This approach, taken separately, allowed some of the models to be transformed into separate software modules for the development phase of ISCT RSL. The following models were created by the authors:
Schemes which are used to identify and demonstrate the aggregate of supporting subsystems in order to ensure full functionality of the developed ISCT RSL and to establish connection between all participants of the educational process. It should be noted that the subsystems were singled out according to the structural features of the subject area that match the ecosystem of the educational organisation in terms of organising and supporting the educational process.
Business Process Model and Notation (BPMN-model) demonstrating the sequence of performers’ actions in working with the information system as well as generation of information flows emerging during this operation. Table 2 shows the main classes of graphic elements used in the developed models, with their interpretation for the maintenance of the ISCT RSL in the ecosystem of educational organisation.
Table 2 - Used graphic symbols in BPMN model.
| Element name | Graphic | Characteristic |
|---|---|---|
| Termination of a process |
|
The process terminates after the last action succeeds |
| Receiving a message |
|
Implementation of the process starts when a message or data are received |
| Message dispatch completed |
|
Implementation of the process is completed by sending a message or data |
| Timer |
|
Shows how much time is needed to perform certain actions within the process |
| Error |
|
Indicates that an error may have occurred during the implementation of some action, and defines a new sequence of actions within the process |
| Task |
|
Elementary action performed within the process |
| Collapsed subprocess |
|
Consistent execution of a set of actions that have latent insignificant details within the current process |
| Messages |
|
Data exchange between process objects |
| Parallel gateway |
|
Splitting the process into several parallel subprocesses or their synchronisation |
| Exclusive gateway |
|
The choice of subprocess depends on meeting the condition. It can connect several subprocesses without synchronising them |
| Non-exclusive gateway |
|
When several conditions are met simultaneously the process is divided into the corresponding number of independent subprocesses |
| Set of input data objects |
|
Needed to complete the task |
| Set of output data objects |
|
The result of complying with a task, that can be stored or passed over for execution of another task |
| Input data |
|
Object required to complete the task |
The model based on the elements of Table 2 creates a clear and unambiguous picture of all interaction stages through the use of a single standardised language. The authors used this approach to ensure transparency of the model content for all project participants regardless of the level of their technical knowledge (Logachev et al., 2021). In addition, we have created an opportunity to simplify XML documentation processes in the future for due development of the project and involvement of new developers.
Using the presented methods, the authors were able to identify the key features of the objects involved in the educational process and transform them into a conceptual model (Figure 2). The model on the first line shows the groups of participants in the ecosystem of an educational organization. These groups interact with each other in different processes, which are shown on the second line. It should be noted that members of the same group also interact with each other. At the same time, both in the first and in the second case, the interaction is carried out using certain software tools. They are shown in the diagram as "satellites" of the process. It is important to understand that these software tools can be used not only within the same process around which they are concentrated. For example, the class schedule system is kept up to date during the implementation of educational processes. At the same time, such a schedule can be part of the website of an educational organization that is involved in external processes. Thus, all processes of the educational organization are interconnected.
When commissioning a new software product, all the features of the ecosystem in which such a product will be used should be taken into account. This ensures a consistent state of all software modules of the implemented system with software products already used in the ecosystem.
For ISCT RSL, the following model of interaction with the elements of the existing ecosystem has been developed, shown in Figure 3.
On Figure 3 shows the ISCT RSL modules and their connection with the processes of the ecosystem of an educational organization. A detailed description of the functionality of each of the modules can be found in the work of Logachev et al. (2022).
Their interaction model is presented in Figure 4.
Source: Logachev et al. (2022).Fig. 4 - Model of interaction with ISCT RSL in the educational organisation’s ecosystem.
Let us consider the additional assignments (or modified traditional tasks) to be performed by the main actors in the ISCT RSL process.
Generator. An external module integrated into the process ecosystem in order to allow automated generation of translation assignments (Sidelev, 2018). The assignment is generated as a set of individual words, phrases or sentences with the content set according to pre-defined parameters. These parameters include, for instance, topics (subject area), length of verbal expression, dialect, etc. The structures thus obtained are saved in the system together with the translation template and are further used for exercises or student certification.
Expert. The expert can check the content of generated assignments and make changes at own discretion, with subsequent saving them in the system. The expert has access to the complete list of topics generated on the basis of used vocabulary. The content of assignments can be viewed at any time, but changes can only be made if no student has proceeded to work. In this case, the assignment can be withdrawn with automated recalculation of the score obtained by the students who have completed it. In addition to viewing and modifying assignments, the expert has the opportunity to review the results of their execution. The report generation system allows to generate statistics by different parameters (topic, date, student, group, etc.), with further export in one of the formats (pdf, xml, csv, txt, doc, rtf). Such a report is stored in the system for a fixed time and is accessible to other experts. It should be noted that the expert cannot edit assignments, generate reports or view assignments at the same time. Once the expert starts the work, he/she has to decide which assignment shall be initiated, and the next one will be launched only after the completion of the selected process.
Student. The student has an opportunity to review e-learning resources and complete translation assignments according to the set control and test points. The checkpoint starting process matches the course settings (either after familiarisation with a certain amount of material or within the time interval determined by the instructor). These settings are customised during the tutorial process; therefore, they are not shown in the model.
Teacher. The teacher performs control of assignments that cannot be unambiguously assessed by the information system. This can be accounted for by technical features of verification (for instance, the resolution of received image does not match the minimum required one, the student’s silhouette has a non-standard position in the frame, etc.), or by objective reasons (for instance, the gesture was made at a wrong angle; significant deviation from the reference model, etc.). This verification is only initiated if a due message was received by the controller. In this case, he/she is provided with access to the conditions of the assignment and the received “disputed answer”. This access is granted for a specified amount of time. If no decision is made, the automatically generated assessment grade is fixed. This is reasonable, since the gesture must be unambiguously understood by the receiving party.
The results obtained in the course of the research are theoretically and practically significant, reliable and reproducible. This is ensured by the correct use of the stated research methods and the analysis of the obtained results in comparison with the other scholars’ findings in related studies.
In our opinion, the analysis of results should be made in three main aspects:
1. Formalisation of the gesture language in order to apply the obtained models in the algorithms of information systems used in the ecosystem of an educational organisation. As a result of the present research, the main attributes and characteristics of RSL were identified. If we compare the approaches to the organisation of this process and the derived results with the findings of other scholars, we can find some similarity. For instance, the works by Kharlamenkov (2021), demonstrate exactly the same approach to identification and systematisation of RSL attributes that are further used as models for software-based transformation. Kharlamenkov (2021), uses the derived attributes for quick navigation across the Electronic reference analytical system “Explanatory Lexicographic Dictionary of Russian Sign Language”, while uses them for recognition and analysis of the gesture graphic image with subsequent linkage to its textual characteristics.
2. The possibility of using the obtained results for creation and integration of ISCT RSL into the ecosystem of an educational organisation in terms of the key aspects of methodologies used in the software life cycle. As a number of authors point out in their studies, any educational organisation uses numerous heterogeneous software products. As Stain (2018) points out in his work, all results obtained in the course of designing information systems or expert systems should be of cross-platform nature or be prepared for further transformation for use by other objects of the educational organisation’s ecosystem.
The research results obtained for this area within a number of research groups represent a basis for further development of software products for the ecosystem of an educational organisation, thus having practical significance. The theoretical significance, as noted in the research essays, lies in the creation of methods for transforming data models as focused on their further use. The data model developed by the authors of this article is no exception. The techniques we have used enabled us to obtain similar results to those in the studies we have analysed, relative to the set of properties and characteristics. The legitimate difference is represented by the models’ semantic aspect.
3. Organisation of processes in the ecosystem of an educational organisation without violation of its rules and not contradicting to the requirements of the national legislation in the sphere of education and science. As Logachev & Chernova (2022a), point out in their research, any use of software products in the educational process should comply with the requirements of regulatory documents. In addition, all information or materials used for educational purposes in information systems should be subject to high-quality expert review; the obtained results should yield adequate and consistent qualitative assessment. For these requirements to be met, coordinated interaction of a number of professionals from educational organisation must be ensured. Information systems used in relevant processes are additional objects of this interaction. National and foreign scientists note the necessity in their works (Boulares & Barnawi, 2022), before designing any information systems, to analyse the ecosystem of the organisation that ordered the software products, and to compile accompanying documentation at the stage of relevant software development and further on. Such documentation not only should comprise detailed technical and organisational requirements for inculcation, configuration and operation of a software product, but also contain accurate and clear flow charts or infrastructure sheets describing the sequence of operations in implementing all ongoing processes during the interaction with the information system. BPMN-models are mainly used for this purpose. Our study is not an exception; a similar model was used for explicit demonstration of control flows.
Conclusions
The digitalization of maximum possible amount of functions has become a developmental priority in recent years, primarily due to the pandemic constraints.
Educational institutions are forced to transfer their staff to remote working mode, develop protocols for the educational process based on remote working mode. This increases the number of software tools in use within the educational organization. In order to maintain their operability and consistent state, a shared information system is required that ensures continuous support of different-format data and reduces the role of human factor in decision-making.
Against this background, ISCT RSL is a tool that not only combines the standard learning functionality of such systems for the most part, but also integrates people with disabilities and professionals concerned about RSL into the educational process. At the same time, the ISCT RSL ecosystem can integrate educational institutions’ specialists implementing training programs for sign language interpreters and specialists of different linguistic schools and RSL centers, as well as translate the educational process into the distance learning format.
It should be noted that the problem of efficient sign language recognition has not yet been resolved due to serious differences in the semantic structure of graphic and sign language. This fact does not allow for unambiguous translation of sign language. Thus it is not so far possible to get a fully automated system that allows full-value sign interpretation supporting real-time communication; however, it is possible to provide support of the basic functions and due specialist training. The undertaken research presents a system and respective process organization which make it possible to develop a ISCT RSL that would combine the elements of standard teaching systems with the sign-language training methods.
The models derived in the present research are accurate and universal. Their content is fully consistent with the subject area and allows for generating accurate instructions for specialists involved in development and inculcation of information systems.
The model providing interaction of specialists organizing the educational process demonstrates interrelation of any assignment with the information system content; moreover, it structures the emerging data flows. As a result, it becomes possible to evaluate the content of the information system modules at the stage of its development, as well as the extent of involvement of the system users in obtaining particular results.
The research uses generally accepted and universal methods necessary for the implementation of modeling and design processes of information systems of any complexity. When using them, the authors did not have any difficulties, including when interpreting the results obtained. The created formal models fully meet both the requirements of design methodologies and the requirements in the field of education for the organization of the educational process. The obtained models served as the basis for the technical assignment to create an information control system for teaching Russian sign language. The models contain intuitive elements both for specialists in the field of creating information systems and for specialists using information and communication technologies in their work. The latter, in particular, include specialists acting in the role of customer.
In conclusion, it should be noted that the resulting models are reproducible by any performer and are implemented using any available object-oriented design or programming tools, as well as database management tools.
