FOCUS

 

Integral evaluation of quality in diagnostic laboratory networks

 

Evaluación integral de la calidad para redes de laboratorios de diagnóstico

 

 

Alfredo Rego¹, Héctor Pérez², Liliena López², Niurka Carlos¹

¹Departamento de Matemática y Programación Aplicada
²Departamento de Control de Calidad Centro de Inmunoensayo, CIE Calle 134 y Ave. 25, Cubanacán, Playa, Ciudad de La Habana, Cuba

 

 

---

ABSTRACT

The Center of Immunoassay of Cuba (CIE) has been dedicated for years to develop the SUMA (Ultra Micro Analytical System) technology, applied in pre- and post-natal screening programs, epidemiological surveillance and certification of blood, organs and placenta. CIE has developed and updated the quality control systems based on clinical laboratory theoretical background of this field, to guarantee the quality performance of this technology in the Cuban laboratory network and in other countries. These objectives have resulted in an informatics support or software named Modular System for Quality Control which comprises an informatics tool intended to evaluate the quality of laboratory networks by combining different types of internal control, such as internal, external and statistical data. They support an integral assessment of quality performance in these labs. The elements of design; development and evaluation used by the Modular System for Quality Control are described here. This system is already installed in the laboratory networks of the Cuban Health System, guaranteeing its safety, reliability and quality.

Keywords: Quality, control, evaluation.

---

RESUMEN

El Centro de Inmunoensayo es una institución cubana dedicada durante años al desarrollo de la tecnología SUMA (Sistema Ultra Micro Analítico), la cual se aplica en programas de pesquisas prenatal y neonatal, vigilancia epidemiológica y certificación de sangre, órganos y placenta. Con el objetivo de garantizar la calidad del funcionamiento de dicha tecnología en las redes de laboratorios cubanos y de otros países donde se encuentra instalada, el CIE ha desarrollado y perfeccionado sistemas de evaluación de la calidad sustentados en los fundamentos teóricos de esta rama del laboratorio clínico, los que se han materializado en un soporte informático o software de aplicación llamado Sistema Modular para el Control de Calidad, (consistente en una herramienta informática destinada a la evaluación de la calidad en las redes de laboratorios, que posee la particularidad de combinar los diferentes tipos de control de la calidad (interno, externo y resultados estadísticos) para realizar un análisis integral del comportamiento de la calidad de dichos laboratorios El presente trabajo describe elementos relativos a la concepción, desarrollo y evaluación del Sistema Modular para el Control de Calidad que se encuentra instalado en las redes de laboratorios de diagnóstico pertenecientes a los programas de salud cubanos y constituye un componente de extraordinaria importancia para su seguridad, fiabilidad y garantía de la calidad.

Palabras clave: Calidad, Control, Evaluación.

---

 

 

INTRODUCTION

Quality control or management consists of developing systems to guarantee productions and services that satisfy or exceed client requirements. Skills and technical knowledge are required to implement it in clinical laboratories. These factors generate additional expenses which can be considered irrelevant when compared to the high economical and social costs of the erroneous diagnosis of some diseases.

Diagnostic laboratories incur measurement errors value obtained and the value accepted as real), these errors are been classified as: random (vary unpredictably in magnitude and sign), systematic (remain constant when carrying out measurements under the same conditions) and gross or evitable (1). A laboratory is considered as protected against all these errors by implementing methods able to detect them. These methods are commonly named Internal and External Quality Controls, respectively.

Herman Steigstra and Rob TP Cansen (2) described the relevance of carrying out a combined analysis, in other words, the importance of analyzing the behavior results of both types of controls for quality evaluation. Currently, software tools as the MultiQC system developed by Randox (3) allow carrying out this type of analysis in the laboratory. However, this becomes a complicated task when trying to evaluate the behavior of all the laboratories, due to the high diversity of the equipment and informatics systems with different formats for storage of information.

Quality evaluations in the diagnostic laboratory network belonging to neonatal, prenatal and blood banks in Cuba are carried out through the Modular System for Quality Control (MSQC), which uses software tools to make an integral quality evaluation by combining different types of analysis (analysis of Internal and Externa Quality Controls, external analysis f the Internal Quality Control and analysis of quality based on statistical data).

MATERIALS AND METHODS

Types of analyses used in the laboratory network and external control distribution methodology

The different kits of reagents (diagnostic kits) produced by the Center of Immunoassay (CIE) are classified according to the results in: qualitative (when the result is expressed in a non quantifiable format, for example, positive or negative) and quantitative (when the result is expressed through a quantifiable value of a given physical magnitude, for example, 50 mU/L). In addition to the production of diagnostic kits for different diseases, the CIE distributes sets of standards for internal and external quality controls.

The laboratory uses the reagents supplied together with the diagnostic kits to carry out the internal controls, while using the reagents sent by the laboratories involved in the program for the external controls, with a frequency of 2 to 3 monthly controls per quantitative assay and 8 for each qualitative control.

Methodology to evaluate the results of external quality controls

Quantitative assays

The analyses of the External Quality Control is carried out by using the variation index (VI) as an indicator of the relative variability of the result reported by the lab compared to the average value obtained from the results reported by all the labs (4-6), which is calculated as follows:

Where:
M: is the consensus value obtained from the values reported by the participating laboratories.
X: is the value reported by the analyzed laboratory.
VI: is the variation index.

Different methodologies are used in the methodologies to obtain the consensus value M. Selection is performed by an automatic software. Depending on the number of participating laboratories in the program, the system determines the use of the mean or the median as consensus value, and the accuracy index (AI) is calculated for each VI value obtained for an external control:

Where: RVC: is a reference variation coefficient specific for each assay.

Since the external control Schedule uses two external controls per month for each assay, an average accuracy index 1(AAI1) is obtained for Control 1 and one AAI2 for Control 2, both results are finally averaged to obtain the AAI.

An average variation index (AVI) is calculated as measurement of inaccuracy, when determining the external control, from the absolute value of each AAI, calculated for each control.

To improve interpretation of results of the laboratory network, it was decided to assign a qualitative evaluation to quantitative AVI values, based on quality index (QI), being classified as follows:

Qualitative assays

Evaluation is carried out based on the diagnostic impact of agreement. The analysis is based on the Quality Index (QI), which reflects the agreement between the results obtained in each laboratory and the expected for the assay.

Calculation is performed by starting at a maximum QI of 100 points, being subtracted 6.25 points per each control with an expected negative result informed as positive and 25 points per each control with an expected positive result informed as negative. Best results are closer to 100, and are classified according to the following scale:

Methodology to evaluate the results of internal quality controls

Results of the Internal Quality Control are evaluated in two ways, first, by using the software installed in each SUMA laboratory (2) (SRS/Strips Readers Software) which verifies that the obtained results for each assay control do not exceed the limits defined in the manufacturer’s quality specifications for the reagent kit. Second, by using the internationally recognized Westgard’s rules (8-10), allowing a more integral analysis.

Westgard’s rules (8-10) are used to define the limits for the acceptability of an assay, starting from the behavior of results by situating them in a control chart or Shewhart chart (8).

The system also offers the possibility to analyze the behavior of the internal controls of laboratories in the network (only applicable to those laboratories reporting their results via e-mail). In this variant, the laboratory-installed system generates a monthly automatic file containing a compressed database with the internal and external quality control information, which is sent by e-mail to the evaluator laboratory for downloading it and detailed analysis of the behavior of such controls.

SMCC software

The SMCC is the software system used as an essential tool for quality evaluation and analysis of laboratories in the network. It is run on Windows operative system and can be configured to function with different types of database managers. It consists of three software systems: 1) SMCC to be used by the evaluator (is installed in the laboratory carrying out the evaluation), 2) SMCC for laboratory use (is installed in all the laboratories of the national network) and 3) SMCC for regional centers (is installed in the regional representations, with four of them in Cuba).

SMCC-evaluator

This is the SMCC installed in the computer of the laboratory network evaluator (SMCC-Evaluator), designed to analyze the behavior of internal and external quality controls of all the laboratories in the network. It also allows configuring the system to work with different types of database managers, comprising six modules: information, statistical functions, analysis of internal quality control, analysis of the lots of reagents, for sending results and to print results.

The module for data receipt supports receiving the data manually or automatically. The information is first introduced manually from printed or telephonically informed reports; in the last case it is not possible to analyze the behavior of the Laboratory Internal Control. In the automatic form, the information coming from the laboratories is automatically received or downloaded into the Evaluator’s System database, and the module calculates the assays quality indicators and automatically selects the most appropriated algorithm to obtain the consensus value.

The module of Statistical functions provides statistical charts with 25 combinations of statistical results indicators versus elements analyzed, such as the behavior of percentage of elevated results (for qualitative results) and reagent results (for qualitative results) of the evaluated assays, the behavior of the AVI and AAI quality indicators, by laboratory, by geographical regions and for the country. It also includes statistics structured as reports, with up to 10 combinations of statistical indicators.

The module for the Analysis of Internal Quality Control supports the automatic evaluation of laboratories from the internal control charts by using the Westgard´s rules (8-10). The evaluator is subsequently aware of a series of recommendations on alerts, alarms and possible causes for the errors indicated.

Additionally, the module for the analysis of the behavior of reagent lots allows studying the quality indicators behavior among different lots, to take the appropriate measures for improving quality during the production process.

The module used for sending results of evaluations of the laboratories on the network generates a file containing a database which contains the results sent by all the laboratories, and it is sent to every laboratory by e-mail or by uploading the file into the website of the company to be downloaded through the system installed in the laboratory.

Additionally, the module for printing the results was designed for laboratories unable to receive information via e-mail or from a website.

SMCC-laboratory

This system is installed in the computer of the network laboratory. It is designed to process internal and external quality controls of all the assays, and comprising modules for introducing additional data, processing the internal quality control, statistical display and to transfer information. The module for introducing additional data supports the addition of information useful for surveillance of diagnostic programs. This information is added to the results of external control measurements (automatically retrieved from the instrument database).

The module for processing the internal quality control also bears manual and automatic modalities. The automatic procedure allows collecting results of internal quality control from the database of the system performing the control of the reading device and results calculation. Data are retrieved from the internal quality control concentrations stored in the local database and processed according to the Westgard´s rules (8- 10). The laboratory can make decisions on the acceptance or refusal of the runs when the possible causes of error are shown. The manual variant supports the introduction of control values, a variant designed to test other controls additional to those included in the kits of reagents.

The statistical display module allows visualizing statistical data as charts, facilitating the analysis of useful statistical indicators’ behavior, which help to infer certain quality behaviors during laboratory work, to improve quality. It also provides options for display of external control results, shown as report sheets.

Finally, the module to transfer information allows receipt and transmission of information among the laboratory, regional centers and the evaluator.

SMCC for regional centers

This system is designed to work with all the technological assays, working as intermediary for transferring information between diagnostic laboratories and the evaluator laboratory.

As previously mentioned, the Cuban territory is divided in four regions for technical and analytical system (East, Center, West North and West South). Each region stores the information related to laboratories in its jurisdiction and once completed, it is sent to the evaluator at CIE. A similar procedure is followed backwards; the evaluator processes the data monthly and sends the analysis results to the regional centers as a compressed file, the centers download it to their respective local databases and send it to the laboratories (sending it printed to those laboratories unable to use e-mail).

The system comprises modules to receipt and send information, analysis of results of statistical indicators presented as charts or printed reports to display them as statistics of laboratories or regions, and the module for analysis of the internal control that supports the behavioral analysis of laboratory quality for those laboratories under the jurisdiction of the respective regional center.

Analysis of the external quality control

Results of the external quality control evaluation for each laboratory are generated as tables, charts and histograms. The table shows the information supplied by the Evaluator System when evaluating the results of a laboratory in a specific quantitative assay (quantification of alpha-fetal protein in maternal serum). It shows the results or reference standards for two external controls C1 and C2, the values reported by the laboratory under evaluation, results for the calculated variation index (VI), the accuracy index (AI), the average variation index (AVI), the average accuracy index (AAI) and the total number of laboratories involved.

The system generates the reference value by calculating the mean for all values reported for each control when the number of laboratory exceeds 20, and eliminates those above or below the mean by three standard deviations.

If the number of laboratories included is between 10 and 20, the reference value is obtained by using the median, and a value assigned by the producer of external controls if lower than 10, those controls being obtained when certifying it at the Department of Quality Assurance and Control.

Figure 1 represents a diagram of results obtained from a network laboratory. The red dot represents the position for the laboratory under evaluation and the white dots represent the results obtained by the rest of laboratories. The inverted V shape observed in most of diagram dots stands for the IEp1 and IEP2 values with the same sign among laboratories represented, what results in IEP and IVP values with the same modular value and their respective dots are represented equidistantly from X and Y axes.

Analysis of the internal quality control

The system evaluates behavior by calculating the statistics represented in the statistical section and generate its alert or alarm warnings, based on complying or not with the chosen rules, and also making recommendations and indicating possible causes for such a behavior.

Figure 2 shows an example of the system’s chart when analyzing the internal control according to Westgard ´s rules; the software uses the most usual rules (1C2SD, 2C2SD, 4C1SD, 1C3SD and 10X). The assay and lot must be selected and press the button to generate the chart. Next, the system calculates the statistics shown: mean (MD), standard deviation (SD) and the variation coefficient (VC), elaborating the chart. It is important to notice that aberrant values (also known as outliers) are eliminated to calculate the statistics. Additionally, the behavior of controls is analyzed by applying the Westgard´s rules and the system informs with possible warnings of alert or alarm. The system also gives a set of recommendations indicating the possible causes of error and how to solve them.

RESULTS AND DISCUSSION

The idea of evaluating the quality of network laboratories by combined analysis of internal and external quality controls was proven for the Cuban laboratory network. The behavior of both controls was studied through the SMCC, confirming that some laboratories were deficient when its internal control was analyzed (according to Westgard´s rules), in spite of obtaining satisfactory and even excellent results in the external control quality evaluations. The opposite was also detected in some cases, although to a lesser extent.

The above mentioned cases sometimes facilitate a considerable increase in the consumption of reagents, subsequently increasing the number of technical assistance visits (either analytical or instrumental) to the laboratories. This also decreases the efficiency of the network function and promotes the emission of inadequate evaluations.

The analysis by using a combined schedule is a distinctive aspect of the SMCC among the vast majority of international quality control schedules where an evaluator executes the evaluation of the external control and the laboratory assumes the internal control analysis. However, a limiting factor to consider is that to do so the system evaluator must receive the file generated by the laboratory, which must be sent by email and laboratories do not always have this possibility.

Finally, it is important to notice that this system has had a significant impact on the increased quality of Cuban laboratories (as reported for other international systems (11). The SMCC is a significantly relevant tool to identify laboratories with deficient results and to carry out the corrective actions required to guarantee the functioning of Cuban health programs.

 

REFERENCES

1. Krouwer JS. Setting performance goals and evaluating total analytical error for diagnostic assays. Clin Chem 2002;48(6):919-27.

2. SUMA. Sistema Ultra Micro Analítico. (Sitio en Internet). Disponible en: http://www.tecnosuma.com. 5 de agosto 2008.

3. Steigstra H, Jannsen RTP, Baaenhuijsen H. Combi Scheme: New combined internal / external quality assessment scheme in The Netherlands. Clin Chem 1991;37(7):1196-1204.

4. MultiQC. Sistema de Control de la Calidad para Química Clínica.2008. (Sitio en Internet). Disponible en: http://www.multiqc.com. 2 de marzo 2008.

5. Whitehead TP. Advances in Quality Control. Clin Chem 1977;19:175-205.

6. Hill P, Uldall A, Wilding P. Fundamentals for External Quality Assessment. Prepared by the Committee on Analytical Quality (CAQ) of the Education and Management Division of International Federation Clinical Chemistry (IFCC). Milano, Italia;Julio 1996.

7. Gibbs WN, Britten AFH. Guidelines for the organization of a blood transfusion service. (Procedimientos de evaluación externa). Geneva, Switzerland: World Health Organization;1992.

8. Westgard JO, Barry  PL, Hunt MR, Groth T. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981;27:493-501.

9. Westgard JO, Barry PL. Improving Quality Control by use of Multirule Control Procedures. In: Cost-Effective Quality Control: Managing the quality and productivity of analytical processes. Washington, DC: AACC Press;1986:92-117. 

10. Westgard JO, Klee GG. Quality Management. In: Burtis C, ed. Fundamentals of Clinical Chemistry. 4^th ed.Philadelphia: WB Saunders Company;1996:211-23.

11. Libeer JC. Role of external quality assurance schemes in assessing and improving quality in medical laboratories. Clin Chem Acta 2001;309:173-7.

 

Received in September, 2008.
Accepted for publication in March, 2009.

Alfredo Rego. Departamento de Matemática y Programación Aplicada. Centro de Inmunoensayo, CIE. Calle 134 y Ave. 25, Cubanacán, Playa, Ciudad de La Habana, Cuba. E-mail: [e-mail]