(pdf)
SciELO 
SciELO 
Permalink
Introduction
Educational research, as an essentially scientific process, becomes a key element in the management of science and technological innovation. Therefore, its study and evaluation are essential and thus approached by several authors (Ortíz, et al., 2009; Ortíz, 2015; Alonso, et al., 2016; Venet & Barros, 2017). This responds to the current concern “that many of the results of educational research do not become a source of changes and educational improvement and, on occasion, for reflection on pedagogical practice”. (Venet & Barros, 2017, p. 420)
In view of this difficulty, evaluating the results of educational research is a need, not only to corroborate its effects and improve the conception and development of these studies, but also to systematize new ways that allow promoting management of its impact from an integrative and coherent perspective. Given these premises, the present work assumes this approach by offering a critical reflection of some epistemological assumptions about the evaluation of training impact on scientific-educational results, and their behavior at the Universidad de Oriente, Cuba. Likewise, a strategy is proposed for its evaluative management, as an alternative to systematize it in the pedagogical practice.
Development
Epistemological referents of the process of result management on educational research, and the evaluation of its educational impact. According to Castellanos (2000), quoted by Venet & Barros (2017), educational research is the “dialectical process of construction of multidisciplinary scientific knowledge about educational reality, consciously oriented and regulated by the scientific method, with the aim to produce certain scientific-technical results that make it possible to describe, explain, predict and transform the object in correspondence with the immediate and prospective problems of the development of education in a concrete historical context”.
Hence, educational research is recognized as a contextualized process directly related to problems of educational quality. Research conducted in this field "aims at changing the educational reality, increasing educational quality, and thereby contributing to human development" (García, et al., 2009). Consequently, this type of research must be planned, organized and controlled, which implies managing its results to grant reliable and rigorous criteria of the certainties of its effect on pedagogical practice.
The evaluation of the impact of research in the educational field is still insufficiently addressed, both from the theoretical and practical point of view, and authors (Ortíz, 2015; Venet, & Barros, 2017) agree that this issue must be addressed as a way to "improve the conception and development of said research, in such a way that it fulfills its fundamental mission: to promote educational practice towards increasingly optimal levels".
An in-depth study of this topic leads to the interrelation of two concepts, which are complex in themselves: evaluation and impact, which require a sufficient level of theoretical systematization. The evaluation refers mainly to the expression of a value judgment, which implies obtaining evidence of the changes that occur in the subjects and in the contexts with a view to improving the quality of educational practice.
Venet & Barros (2017), synthesize the following peculiarities of evaluation: evaluation as a dialectical, complex and totalizing process; evaluating is obtaining evidence of the changes that occur in the process being evaluated; recognizing evaluation as a process of synthesis and interpretation of change evidence; to evaluate is to use the information obtained in order to improve practice and to evaluate is to contribute to monitoring and improving the quality of educational practice.
This conceptual position reveals the essence that characterizes the evaluative process, from its integrating nature, since it results from a perspective of conceptual approach that does not exhaust the evaluation of the research result by itself, but rather dialectically combines the evaluation of the finished product with the monitoring and development of the process.
The category “impact” presents a number of interpretations that are still discussed at an international level from different perspectives and conceptions. Blankenburg (1995); cited by Cruz & Martínez (2014) has indicated that the impact is framed in the “broad and sustainable changes introduced by a given intervention in the lives of the beneficiaries. The impact can be related to the specific objectives of an intervention or to unexpected changes caused by an intervention; Such unexpected changes can also occur in the lives of people outside the target group. The impact can be positive or negative”.
On the other hand, in shorter way, Wainwright (2002); cited by Cruz & Martínez (2014) states that “it is any change resulting from an activity, project or organization; and includes the desired and undesired effects, positive and negative, brief and lasting” (p. 2). According to Cabrera, & Herrera (2003), cited by Tejeda (2011), one of the meanings of the term is: set of changes, results and effects, which can be positive and negative, since it includes both in relation to the system of reference or previously selected standards.
In both positions the impact can be unexpected or unpredictable, even negative, which connotes these definitions from a too general vision, therefore reducing the effectiveness and uniqueness of the results of the scientific-educational activity. Hence, it is convenient to approach the impact from a more optimistic and lasting perspective, without ignoring "the possibility that a result may have a negative impact on the object for which it was conceived, or on another one that was rejected during the preliminary forecasts". (Cruz, & Martínez, 2014, p. 2)
The common ground to these definitions of impact is recognizing it as the benefit achieved, verifiable and measurable, expressed in objective indicators, with the perception of favorable, sustainable and relevant changes in the economy and society, obtained by the application of the results. Scientists (Villaveces, et al., 2005).
Based on these general definitions, it is possible to sustain the unique and transformative essence of educational research, with educational change among its fundamental purposes. Therefore, the concept of impact of educational research provided by Venet, & Barros (2017), is assumed, understood as “the set of relatively enduring educational transformations that occur in institutions and subjects that are involved in this process, under the influence of the partial and final results that are obtained by means of the investigative scientific way, defining a superior situation with respect to the initial state”. (p. 425)
Consequently, it is essential to improve the evaluative management of impacts of educational research, an important element to assess the quality of research results in this field. This is defined which from the perspective of this study as an intentional process that promotes the integration of educational practice with academic training and research and that includes the planning, organization, implementation and control of changes in subjects and socio-educational contexts.
Therefore, it becomes a tool that allows to foresee and evaluate expected changes as a result of the development of scientific-educational activities, since it is not possible to understand impact as a spontaneous consequence of the results achieved, they must be designed proactively, so that impact is confirmed from the first moments. It is the evaluation as a continuous and regulatory process that allows to determine the effectiveness of the results (Cruz & Garcia, 2013).
Due to the very nature of educational research, educational impacts are recognized as one of the most essential and visible elements to promote and evaluate the changes that occur in this area. This is understood as the improvement of education, aimed at moving towards a more intelligent, sustainable and inclusive growth, which must be produced with the highest scientific rigor (Flecha & Álvarez, 2015). Therefore, it is considered as the effect produced in the subjects and the process that indicate gradual transformations, from the reciprocal influence that is generated, from the dynamics of the various instructive and educational variants used in a certain period of time, according to the intended development intention (Tejeda, 2011).
This conception of formative impact, allows to recognize its evaluation as an assessment process on the sustainability and improvement of educational results, which leads to consider it as an investigation of evidence and assessment of the effect produced on the subjects and the process, as a sample of the effectiveness of the actions and variants designed and applied in a training context and a specific period, in accordance with the budgets and planned purposes (Tejeda, 2011).
This author proposes four levels of training impact evaluation for their application in a specific educational context, where the scope given by the managers and the level of intervention that has been conceived to develop the inquiry and assessment of the effect stands out. They are:
Level of evaluation of the reaction of the training act: the aim is to develop a search for the satisfaction of the subjects according to the training, based on the correlation between training needs and personal expectations.
Level of development reached in the training subjects: it is characterized by an inquiry into the gradual changes that have occurred in the subjects (students and teachers) from the influence of the actions and training variants applied in the context.
Level of transfer achieved by the participants: assesses how learning through the various training activities has been applied in one or more contexts, connoted by the potentialities and opportunities of the subjects involved.
Level of changes produced in the performance contexts of the subjects: assesses learning durability through one or more training variants. It combines the level of development reached by the subjects, their transfer and personal improvement by helping to develop changes in the solution to situations or problems associated with the processes in which they work. This process is also marked by the improvement of the alternatives and edges of the functions and activities developed.
This study also offers a methodological procedure structured in three interrelated stages, containing actions, guiding questions and integrating aspects that facilitate the evaluation of the training impact. However, the logic defended by this author does not clarify, sufficiently, how to articulate the evaluative levels through the referred stages, nor are concrete indicators required to measure the effects achieved in the educational context.
Managing the educational impact of scientific-educational activity, therefore, continues to be a pending subject for theoretical-practical studies, as it becomes a necessary process to ensure the improvement of pedagogical purposes. This is due to the fact that their quantitative and qualitative assessment is complex, since they respond to processes that need a certain long-term maturation, in order to then evaluate the benefit of their actions on the subjects to whom they have been addressed. This justifies the current imperative to analyze how to evaluate it, so that its scope can be assessed in correspondence with the levels of evaluative intervention already referred to. This aspect is a motivating element of this work.
Consequently, recognizing that the impact on this type of research is legitimized through the educational changes achieved with the introduction of its main results in the context, supposes projecting these changes in advance in the training praxis so as to guarantee an increase in the indicators that account for the quality of the educational process favored with the introduction of the scientific results.
To continue delving practically on how these theoretical limitations revealed in concrete training practice behave, an analysis of result management on educational research and its impact evaluation has been conducted in the Center for Pedagogical Studies "Manuel F. Gran” (CEPED) in Universidad de Oriente, Cuba.
Empirical methods and techniques have been employed to this end, which include: surveys of 20 teachers, 28 current applicants and 20 graduates of CEPED academic programs, interviews with 1 manager, 2 responsible for Science and Postgraduate studies and 5 coordinators of academic and educational programs. research projects (annex 2), as well as the documentary analysis of reports from the Balances of Science, Innovation and Postgraduate Studies, Institutional Self-evaluation, 10 master's theses and 50 doctoral theses and review of project files developed at CEPED over the years 2015 to 2018.
Likewise, scientific observation of training activities of current academic programs was used (defense courts for doctoral theses, master's degrees, seminars and scientific attestations, etc.), which allowed an integrated assessment of the behavior of the process from the definition of the following indicators: main contributions of educational research in relation to training needs, degree of introduction of educational results in the context and educational impact of educational results in correspondence with the needs of the context.
The integrative analysis of the instruments applied revealed the following limitations in result management of educational research and its educational impact evaluation:
The main contributions of educational research are specified in theoretical-practical instruments (models, strategies, methodologies, methods, procedures, etc.) that contribute to the improvement of university processes in correspondence with training needs; However, there is still a limited systematization of the impact management of these training results in educational practice from a logic that integrates the indicators and intervention levels for a better articulation of this process.
The identification of precise indicators that allow operationalizing, in a more objective way, the criteria for the evaluation of training impacts of educational research in the context is still limited, which affects the level of introduction and generalization of its results according to the training needs detected.
The control and monitoring of the educational impact of the research results of the academic programs of the Center for Pedagogical Studies for the improvement of training in university areas and contexts is still limited, which accounts for an insufficient level of evaluation management from the scientific-educational research process itself.
These elements, therefore, allow corroborating that the fundamental causes of the inadequacies in result management on educational research are found in the limited theoretical-methodological conception of this process, which reduces its introduction and generalization in the educational context. Therefore, the need for its reinterpretation is recognized, and it allows to reveal a logic of the management of the formative impact of the results of educational research to guarantee its transformative effectiveness in the educational context.
Management strategy for the training impact of results of educational research
The proposed strategy becomes a relevant alternative that allows reorienting the management of the training impact of the results of educational research, through training actions for its development. It offers researchers a way to achieve the scientific results that are generated in educational scientific activity, evaluating the transformations achieved with its application. It is, in turn, containing stages that allow integrating, in a single logic, the training impact indicators and the levels of intervention to guarantee the transformative effectiveness of scientific results in the educational context.
This instrument is conceived as an open and flexible system, which can be modified based on the changes in the scientific-educational research process. It has a dynamic structure made up of interrelated and articulated actions that favor concordance relationships between its stages, which confers scientific-methodological coherence necessary for its implementation.
Henceforth, the general structure of the strategy for managing the training impact of the results of educational research takes into account (Tardo, et al., 2019):
Assessment of the impact of scientific-educational results and their management.
Premises.
Requirements.
Overall objective.
Methodological guidelines for its implementation.
Stages, specific objectives and proposed management actions.
Evaluation system.
The design of this strategy starts from a contextual assessment of the impact of the scientific-educational results and their management. This allows an integrative analysis of the system of influences that determine the regularities of the management under improvement, and that condition the premises and necessary requirements for its conception, completion, and evaluation.
This contextual assessment allows to determine the premises for the developing circumstances, both favorable and unfavorable, that condition the conception and implementation of the strategy. They are specified in:
A doctoral training program that favors, through its compulsory and optional contents, the management of educational results based on scientific research.
Management of pedagogical practices by teachers and researchers in correspondence with the demands of the educational contexts.
Insufficient management levels by teachers and researchers, of the scientific results of educational research, which is expressed in the limited changes achieved in the subjects and contexts from the introduction of these results.
Possibility of projecting training impacts and systematizing evaluation levels and indicators during the doctoral training of the teachers and researchers to guarantee a pertinent management of their scientific results.
The requirements, on the other hand, are based on the contextual assessment conducted, as the intention that guides the development and application of the strategy, and an essential condition for its existence and improvement. These are:
Transformative intention of the proposed evaluative management, which guarantees a permanent training of teachers and researchers to improve the conception and educational impact of the results of their scientific research.
Diagnosis of the potentialities of teachers and researchers that takes into account their scientific and educational interests and professional expectations to project the actions of the strategy.
Mastery of the scientific research methodology by the teacher-researchers and facilitators of the process, to ensure the coherent development of the proposed evaluation management.
Awareness and responsibility of the stakeholders of the training process (teachers-researchers and facilitators) to accept changes in their research management, from an intentional, proactive and transformative vision of their scientific-educational results, as an essential training quality to be developed.
Appropriate pedagogical atmosphere for exchange and scientific commitment of the subjects involved with their educational contexts of performance. This implies changes in the management of their formative impacts from the resignification of contents, methods and the systematization of their experiences in the introduction and generalization of its results according to the specific needs detected.
Its general objective is: to systematize the management of the formative impacts of scientific-educational results of teacher-researchers, based on indicators and levels of intervention to guarantee their transformative effectiveness in the educational context.
The strategy is structured based on the integration of four stages, which are specified in their corresponding system of training actions. These will be implemented at different moments by the teacher-researchers, not in a dogmatic or mandatory way, but as alternatives that to be transformed to guide the evaluative management proposed.
Prior to the application of this strategy, the following training management actions are suggested:
Initial exchange with teacher-researchers and other stakeholders in the training process, to socialize the theoretical-methodological foundations that support the strategy.
Reflection workshops with all the subjects participating in this training process (teachers-researchers, members of the faculty, professors and supervisors of CEPED academic programs to develop the methodological instructing process that guarantees the preparation of researchers and applicants in training.
Definition of the accountabilities of the teacher-researchers, faculty members, professors and supervisors participating in the development of the strategy.
Planning of scientific-methodological workshops about the process of evaluative management of the formative impact of scientific-educational results, whit the following expected achievements:
Establishment of a single management logic that allows the subjects involved to develop their training potential to project their training impacts, manage their training needs, systematize the levels of evaluative intervention and assess the impacts in the educational context.
Encouragement of the motivation of teacher-researchers in the impact management of their scientific results, based on the recognition of their potential and effectiveness in the educational contexts of application.
Familiarization of teacher-researchers with experiences in managing training impacts in diverse educational contexts, which favors inquiry, argumentation and search for solutions to the problems diagnosed.
Design of training objectives to guarantee the systematization of specific content and methods for managing the training impacts of educational results on teacher-researchers.
Four stages are proposed in the strategy, each one with its corresponding specific objectives and specific training management actions. These are:
Stage of proactive conception of the expected training impacts.
Stage for the management of training needs.
Stage for the systematization of evaluative intervention levels.
Stage of evaluation of impacts in the educational context.
Stage of proactive conception of the expected training impacts
Specific objective: To plan the possible educational changes expected as a result of the development of scientific-educational activity.
Actions to be developed by the teacher-researcher
To review permanently the epistemological and methodological logic of the research, projecting the possible changes to be achieved.
To determine the most relevant educational contexts for the application and evaluation of the changes.
To analyze permanently the new epistemological relationships intended as contributions, and their concretion potentialities in the selected educational practice.
To conceive the expected impact on professionals (teachers, students or other training managers), in their performance and in the educational situations in which they are involved.
To reflect on possible indicators to be used to project the possible transformations in the subjects and the educational practice, that allows guiding the research actions to be developed
To assess the application relevance and transformation of the results during the research process, reevaluating possible adjustments and changes according to the educational problems detected.
Stage for the management of training needs
Specific objective: To identify and analyze emerging problems, needs and potentialities of the subjects who participate in the investigated educational contexts.
Actions to be developed by the teacher-researcher:
To select relevant tools or instruments for data collection.
To identify the training needs, categorize them and agree on their priority levels, taking into account a detailed analysis of each revealed need.
To identify the most essential difficulty, weighing not only the limitations but also the strengths, potentials and opportunities (Deroncele, & Gross, 2018).
To identify the causes that could influence both the current situation and the desirable one, in terms of existing and required conditions.
To make visible the desirable and transformed situation, also in terms of possible results.
Specify on the most essential and emerging difficulties and the possible expected impacts.
To determine the potentialities existing in the educational contexts under research, as well as the resources and possibilities of the subjects involved that can be used to guide the changes.
Stage for the systematization of evaluative intervention levels.
Specific objective: To consolidate, gradually, the scope and durability of the impacts achieved in the educational context from levels of intervention.
Actions to be developed by the teacher-researcher:
To select the beneficiaries of the intervention, the subjects responsible for its application, the specific evaluation context, and the specific moment for its implementation.
To evaluate the relevance of the results provided in correspondence with the potentialities and limitations of the specific training processes of application.
R. To review frequently and systematically the scientific-educational results implemented. This will allow establishing the conditions required so that the emerging impacts are not verified at the end of the investigative process, but are generated as a permanent and intentional condition of the gradual transformations that are achieved as part of the planned training actions.
To specify the levels of evaluative intervention, and the indicators that allow evaluating the changes as far as possible.
Stage of evaluation of impacts in the educational context.
Specific objective: To evaluate and control the educational impact achieved with the introduction of scientific results in educational practice, revealing the specific transformations achieved.
Actions to be developed by the teacher-researcher:
To define evaluation criteria and indicators in the short, medium and long term, which allows to demonstrate the suitability of the scientific research deployed and the effectiveness of the results that are proposed.
To compare the specific changes achieved with those planned and projected at the beginning of the investigative process, according to established indicators.
To specify and evaluate the scope of the changes obtained with the application, to establish which are more tangible from a quantitative and qualitative perspective.
To assess the evidence identified in the evaluated training context (subjects and process) that allows to confirm the real effects produced with the application of the scientific-educational results (Ramos, et al., 2016).
To maintain permanent monitoring and control of the investigative process so that the actions and results introduced in the specific educational contexts can be perfected.
Control and evaluation system for the strategy:
Objective: To assess the qualitative changes in the evaluative management employed by the teacher-researchers, by means of the precision of the efficiency level of the actions conducted in the proposed stages.
The strategy be flexible, as its control and evaluation will be carried out systematically, allowing the pertinent adjustments: as the planned actions are being conducted, or others appear that enhance their implementation. Therefore, the evaluation is conceived from comparing the systematically obtained information and the evaluation of the effectiveness of the planned actions. This implies that the evaluation starts from the socialization of the strategy, considering the motivation achieved in this regard and the disposition of the subjects involved for its implementation, which is why it does not constitute a final stage in its implementation, but rather a dynamic element of each of its stages.
The indicators for evaluating the impact of this strategy, aim at achieving the following transformation levels:
Level of satisfaction of teacher-researchers with the training received.
Verifiable indicators:
Commitment and motivation of the subjects involved with the content received.
Interest and willingness to manage the educational impacts of the results of their educational research.
Level of development reached by teacher-researchers during their training.
Verifiable indicators:
Gradual changes in the subjects (teachers-researchers, faculty members, tutors of academic programs) with the application of the strategy, expressed in:
Levels of argumentation of the indicators, ways and instruments to be used for the evaluation of the formative impact from the singularity of each research.
Quality and level of argumentation of the management proposals of the formative impacts of their presented research.
Visible impacts on the improvement of educational practice (Qualitative and quantitative transformations revealed in the teacher-researchers, subjects and educational practice in the research; evidence of the application of the results and impact achieved: photos, videos, introductory endorsements, socialization , generalization, applied instruments and their processing; recognition of specialists, scientific commissions of academic programs and courts of defense of the validity of the impacts revealed; proposals for new training programs on the management of training impacts in academic programs, etc. ).
Conclusions
The theoretical and methodological analysis hereby conducted has revealed the existence of insufficient theoretical-methodological references to the educational research management process, since it has not yet been possible to systematize, with an integrative logic, the evaluation of the scientific-educational results in relation to managing its impacts. Therefore, the need for a re-interpretation of this process is recognized, and it allows to reveal a logic for the formative impact management of the results of educational research to guarantee its transformative effectiveness in the educational context.
The existing inconsistencies in the process of managing scientific-educational results and the evaluation of their impact in the Center for Pedagogical Studies "Manuel F. Gran" (CEPED) in Universidad de Oriente, Cuba, are corroborated. These are based on insufficient training of the researcher to manage its impacts, while defined criteria are not yet revealed around how to plan, implement, control and evaluate them in the logic of scientific research.
A strategy for managing the educational impact of the results of educational research is proposed as a training alternative to improve the contribution of these research to the demands of the educational context. This instrument supports a coherent theoretical-methodological articulation, from revealing, in an integrating movement, the logic of systematization of indicators and levels of evaluative intervention for a pertinent scientific training of teacher-researchers.
