4 курс / Лучевая диагностика / ВОЗМОЖНОСТИ_СИСТЕМ_АВТОМАТИЧЕСКОГО_АНАЛИЗА_ЦИФРОВЫХ_РЕНТГЕНОЛОГИЧЕСКИХ

89

CHAPTER 5 EFFECT OF X-RAY AUTOMATED ANALYSIS RESULTS ON

HOW RADIOLOGISTS MAKE DECISIONS WHEN WORKING TOGETHER

At this stage of the research, we have studied possible options for the introduction of automated analysis systems of digital X-rays as a method for detecting pulmonary nodules and masses in the clinical practice of radiologists.

At this stage, 20 doctors right after completing their residency in radiology were asked to analyze 100 chest X-rays: 94 X-rays with no pathology and 6 X-rays of persons with confirmed pulmonary nodules and masses (Sampling Package 2 of 100 X- rays with the 94%:6% norm/pathology ratio).

The radiologists were divided into two groups (Group 1 and Group 2), 10 persons each, in a random manner. Persons in the groups have the same work experience.

Testing of Group 1 consisted of two stages.

At Stage 1, Group 1 (n=10) evaluated chest X-rays from Sampling Package 2 on their own; then at Stage 2, the radiologists again analyzed the same X-rays from Sampling Package 2 only supplemented with the X-rays results from an automated analysis system. When analyzing X-rays, the specialists had to fill in a form with answers, by assigning the norm/pathology status to X-rays.

Testing of Group 2 (n=10) consisted of one stage, during which the radiologists received X-rays results from an automated analysis system together with initial chest X- rays from Sampling Package 2. During the testing, the specialists had to fill in a questionnaire where they had to indicate a category for an X-ray – norm or pathology.

At different stages (depending on the group the radiologists were assigned to), they were asked to have a look at the analysis results (norm/pathology solution and a

‘heat map’ with approximate pathology localization), as shown by Program A with this package.

The decision to select this program for the test was due to its highest values of diagnostic efficiency and compliance with the necessary quality criteria of the model

91

As we can see, during Stage I of the testing the sensitivity value at the initial examination by a radiologist was 66.7%, the specificity value was 89.4%, which was significantly lower than the results shown by the automated analysis system (sensitivity of 83.3%, specificity of 99%).

At the Stage II of the testing, during the subsequent analysis of digital chest X- rays supplemented with the interpretation results of images by the automated analysis system, the sensitivity value remained the same and amounted to 66.7%, the specificity value made up to 95.8% (by 6.4%). Therefore, upon reviewing the results of the automated analysis and repeated norm/pathology decision, the diagnostic efficiency increased (Figure 39), but failed to reach or exceed the results shown by the automated analysis system of images.

Figure 39: Testing results of Group 1 of radiologists

The greatest difficulties were from by the interpretation of the chest X-ray in Case N1 – the pathology was localized in the right upper pulmonary lobe behind the shadow of the 1st rib; this pathology was missed by the automated analysis system of digital X-rays and by three radiologists; thereat, when analyzing the X-rays again

93

as an image with pulmonary pathology, the radiologist who missed the nodule at Stage I of the test did not change their mind and classified the image as norm.

To summarize the above, the under-diagnosis value among radiologists decreased from 36.7% to 33.3% after a repeated reading of X-rays supplemented with the analysis results provided by the automated analysis system; at the same time, the over-diagnosis value decreased from 16.1% to 5.4% after a repeated reading of X-rays supplemented with the reply provided by the automated analysis system.

Testing results of radiologists from Group 2 are given in Table 21.

According to the X-ray analysis results in Group 2, the sensitivity value was only 58.3%, the specificity value was 94.7%; namely, it was less than in Group 1 and significantly less than when analyzed by the automated analysis program (Figure 40).

Figure 40: Testing results of Group 2 of radiologists

The greatest difficulties in the analysis of X-rays were also with Cases N1, N2 and N3. The X-ray in Case N1, also falsely interpreted by the automated analysis system as norm, was missed by seven out of ten radiologists. When analyzing the X- rays in Cases N2 and N3, correctly identified as images with pathology by the automated analysis system, seven and six out of ten radiologists falsely interpreter the images.

The least omissions of pulmonary pathology among radiologists from Group 2 were when analyzing X-rays of Cases N5 and N6, similar to the testing results Group 1. The X-ray of Case N5 was correctly identified by all ten radiologists as pathology, by

95

Currently, the most justified tactic in detecting pulmonary nodules and masses during screening X-ray examinations (i.e., with patients without complaints) is to further examine patients using computed tomography.

Based on the X-ray interpretation results provided by the automated analysis system, we had to send six people to undergo CT (five truly positive and one truly negative) and according to the screening results, only one pathology was missed and only one extra CT scan was performed.

The worst results were obtained when interpreted by radiologists from Group 2; before analyzing the images, they received interpreting results provided by the automated analysis system together with the X-rays.

In this case, both the number of possible pathology omissions (ranged from 0 to 4, the arithmetic mean of 2.6; the median value of 2.5) and unjustifiably performed CT scans (ranged from 0 to 19, the arithmetic mean of 5.7; the median value of 5) increased in number.

The results were slightly better when radiologists first independently analyzed the X-rays, and then additionally interpreted them by reviewing the results of the automated analysis system. In this case, both the number of possible pathology omissions ranged from 0 to 4 (the arithmetic mean of 2; the median value of 2), and unjustifiably performed CT scans ranged from 1 to 13 (the arithmetic mean of 5.1; the median value of 4) increased in number.

As shown by the analysis presented during the joint interpretation of the X-rays by a radiologist and an automated analysis system, there is a summation of human and system’s errors, which leads to a deterioration of diagnostic efficiency (sensitivity reduction from 83% to 56.7%, specificity – from 99% to 93.9%), and good results of analytical validation from the automated analysis systems do not correlate with the results of clinical validation.

The use of a model in which images are initially evaluated by a radiologist independently, followed by their evaluation with the help of interpretation results provided by an automated analysis system and subsequent repeated decision-making is

97

CONCLUSION

To create a model of X-ray screening to evaluate the informative value of digital PA X-rays in detecting pulmonary nodules and masses based on the qualifications of radiologists, two databases of X-rays were developed and registered; they consisted of digital chest X-rays and chest CT scans with the further creation of three sampling packages based on these X-rays to test radiologists and automated analysis systems of X-ray images. There were two testing options for the radiologists: in-person testing for 75 specialists and online testing through an online platform for 516 radiologists. Four automated analysis systems were chosen for the test, based on the following criteria: availability of a computer program registration certificate/patent; availability of a test online access; availability to detect pulmonary nodules and masses described in the software’s details. Also, possible options for the introduction of automated analysis of digital X-ray images into the clinical practice of radiologists were reviewed by testing with simulating two different situations of interaction between a radiologist and an automated analysis system of digital X-rays.

Work experience did not affect the sensitivity values of radiologists. The percentage of correct answers was almost the same for the radiologists with less than 10 years of work experience (72.6%) and for the specialists with more than 10 years of work experience (71.3%), while the average percentage value of detected pathology was higher among doctors with less than 10 years of work experience and amounted to 72.6%. Whereas the highest average percentage value of missed pathology was among doctors with more than 10 years of work experience (28.3%).

During the in-person testing of radiologists, the influence of experience in thoracic radiology on the quality of interpretation of digital chest X-ray was also studied. The specialists were divided into two groups: radiologists exposed to thoracic radiology (N=11) and radiologists with no such exposure (N=64). The research revealed no difference in the values of diagnostic efficiency among doctors with and