3-Dimensional Automatic Segmentation of Pituarity Tumor using Deep Learning

Background: The development of a benign pituitary tumor progresses very slowly. Due to this slow development, it may take time to diagnose the patient. The tumor that will form in the Pituitary Gland, which is effective in the secretion of many hormones and located behind the optic nerves, may cover 2/3 of the Pituitary Gland. At the beginning of the Pituitary Tumor, the patient's complaints are not obvious. As the pressure on the nerves increases due to the growth of the tumor, some symptoms appear. In people where hormonal balance is important, due to Pituitary Tumor; Cushing's syndrome diseases can be seen as a result of irregular menstruation, visual disturbances, headache, imbalance in breast milk production, excess ACTH production. Cushing's disease is a fatal disease that can also be triggered by the adrenal glands. Excess ACTH can lead to excessive weight gain, the appearance of fragile bone structure, skin scars and emotional changes.

Materials & Methods: Pituitary Tumor is located in the deepest part of the brain and it is very difficult to perform a surgical operation there. Tumor surgeries, which are obligatory, must be meticulously planned. There is a technique called Transsphenoidal, which is made by entering the posterior bone of the nose. Since this technique requires serious expertise, it is performed by a limited number of experts and centers. Semantic segmentation using deep learning techniques can be quite successful.

Results: With our study, automatic segmentation of the tumor with IoU score up to 98% was possible. This success achieved is quite high and promises hope for the CAD system to be created for Pituitary Tumor.

Conclusion: The 3D-Unet technique, which has been developed in recent years, can perform automatic segmentation in 3 dimensions. In this study, it is aimed to automatically segment a Pituitary Tumor, which requires a difficult operation, with the 3D-Unet model. The Computer Assisted Diagnosis (CAD) system to be created with the method will provide an objective assistance to the specialist from the planning of the operation to the follow-up after the operation.

Hormonal structure is one of the important systems for the human body to perform its duties in a healthy way. Thanks to hormones, the body works in a clear balance and order. The most important organ of this structure is the pituitary gland, which regulates the secretion of hormones while secreting many hormones and ensures that they work effectively (Simander et al., 2023; Geer, 2023). The pituitary gland is in the middle of the lower part of the brain, between the nasal root and the base of the brain, the optic chiasm and optic nerves, where the optic nerves meet in the upper part, the thalamus, which is called the brain of the brain, on the upper sides, the carotid arteries, which are two of the 3 vessels feeding the brain, and the 3 vessels feeding the brain in the back. and the brain stem, which is the respiratory and cardiac control center. Tumor development in this pituitary gland because it is a secretory gland and developmental anomalies are common because it develops from brain palate cells. Depending on these diseases, due to the anatomical structure and neighborhood, vision problems by compressing the optic nerves, problems in cerebral blood supply due to pressure on the carotid vessels, and brain dysfunction due to its enlargement into the brain may develop (Sharma et al., 2023; Ciavarra, 2023). Growth retardation and sexual development problems occur as a result of the incomplete development of hormones due to developmental defects. As a result of pituitary gland secretory hormone tumors, serious life-threatening hormonal disorders that impair body functions may develop (Alkan et al., 2022).

The general and effective treatment of these problems in the pituitary gland is surgery. There are 2 methods to reach this area, which is difficult and risky to reach due to its anatomical location. The first is the transcranial method, in which the pituitary gland is reached by opening the skull and traveling a long distance through the brain tissue, main cerebral vessels and optic nerves. The second method is the transsphenoidal method, which is entered through the nasal cavity and opened the sphenoid cavity and reached the pituitary gland. It is very preferred because of the opening of the brain with the transcranial method and the inability to fully control the pituitary gland and the inability to perform an effective surgery. The transsphenoidal method is more preferred to reach the pituitary gland more easily than the transcranial method, due to better control of the pituitary gland and less damage to the tissue during surgery (Tsuneoka et al., 2023; Trimpou et al., 2022). Complication risks are high in this method due to its anatomical location and neighborhood. Complications and rates that may be encountered after transsphenoidal surgery, taken from https://www.turknorosirurji.org.tr (2020), are given in Table 1. In order to perform an effective surgery in this area, which has many anatomical variations between people, it is necessary to determine the structure and shape of the tumor in order to completely evacuate the tumor, to determine the transsphenoidal anatomical path well for a successful surgery with little damage to the tissues, and to determine its relationship with important structures in the neighboring areas in order to reduce the risk of complications. While the operations are performed in a 3D environment during the surgery, preoperative planning is done through 2D imaging and it cannot reveal an effective anatomy, thus reducing the success rate in surgical planning and during surgery. As a result of better revealing both the surgical area and the surgical path and neighborhoods with 3D imaging, the success of the operation will increase, there will be less tissue damage, and the risk of reoperation and complications will be reduced by ensuring that the tumor is evacuated more effectively.

Table 1: Post-transsphenoidal complications

Pituitary tumor surgeries are very risky surgeries. The more information about tumor size, location, proximity to optic nerves and thalamus, and closeness to the carotid artery to perform this surgery, the more successful it will be and the lower the incidence of postoperative complications. In the current Magnetic Resonance (MR) imaging methods, 3D tumor imaging cannot be performed, and blind spots may remain in the endoscope devices used during the surgery. It will be very useful if you have information about the 3D image of the tumor, carotid artery and optic nerves before the surgery.

Deep learning methods are very successful in processing medical images compared to classical methods. In 2D segmentation, the specialist doctor must select the appropriate slices from the MR image slices. In addition, the image obtained as a result of image processing will be 2 dimensional. 3D U-Net is a new way of deep learning. This method can do the training by using much less images with a lot of images that other deep learning methods need (Altun and Alkan; 2022c). Considering the difficulties of providing medical images, the method is a great gain for the medical field. Since the method reduces the classification and then segmentation down to the pixel, it can give higher success as can be seen from the studies in the literature (Egger et al., 2016). The most important unique aspect of the project is that it will segment the pituitary tumor with a method that has not been studied before, such as 3D U-Net.

In this study, 3-dimensional automatic segmentation was performed using contrast-enhanced T1 sequence MR images from women diagnosed with Pituitary Tumor. Our study was limited to female patients, and the study focused on women in this disease, which disrupts the hormonal balance.

Ali Mohammad Alqudah et al. conducted a study on the segmentation of meningioma, glioma and pituitary tumors by deep learning method. They obtained the data set they used in their studies from the website “https://figshare.com/articles/brain tumor dataset/1512427/5”. A total of 3064 T1-weighted MR images are included in this database. The number of images differs for the three different tumor types. They used 70% of the data set for training and 30% for testing. The study was studied differently with CNN deep learning method for 3 different sections using original images, images obtained by cropping the tumor region, and images obtained by tumor segmentation. They created the CNN architecture themselves and it consists of 19 layers. As a result of the procedure, the average classification success was calculated as 98% (Alqudah et al., 2019).

Jan Egger and his friends state that since the pituitary gland is the manager for the secretion of hormones, which is an indispensable part of the body, an abnormality that occurs here affects the body negatively. They cite Cushing's disease, which is caused by a pituitary tumor as an example. In addition, it is mentioned in the article that tumors larger than 1 mm may require surgical intervention, while tumors smaller than 1 mm can be treated with various treatment methods without surgical intervention. They used T1 and T2 weighted MR images of 10 patients in their study. The tumor volume obtained by manual segmentation in the open access 3D Slicer program and the segmentation success of the GrowCut interface of this program were compared by looking at the DICE score. They calculated a DICE score between 75% and 85% (Egger et al., 2013).

Song et al. mention that since there is no 3-dimensional imaging in the endoscope devices used in pituitary tumor surgeries performed with the transsphenoidal method, vital organs such as optic nerves and carotid artery can be damaged during this surgery. The aim of the study is to design a new endoscope device that displays the pituitary tumor, optic nerves and carotid artery by using computed tomography (CT) images and gives an audible alarm when the surgeon approaches these organs during the operation. Computed tomography images were segmented using 2D U-net deep learning method. In the pre-processing part of the study, the tumor, optic nerves and carotid artery were manually segmented by the specialist using the 3D Slicer program in all 1218 slices of CT images of 6 patients used for training. By preprocessing 11 augmentations from each slice, the number of images was extracted 13398. The U-net operation is divided into two parts. In the first part, 1218 images and labels were trained. In the second part, after 20% of the images obtained by augmentation were reserved for the test, the training was continued by using the weights obtained in the first part. It is stated that the training process takes too much time and it is aimed to reduce it as the reason for the two-stage system. In the Jaccard test they used to determine the segmentation success, they achieved a success rate of up to 0.76 for the optic nerve, 0.72 for the carotid artery, and 0.75 for the tumor (Song et al., 2019).

Victor E. Staartjes et al. emphasize that the transsphenoidal method is frequently used in pituitary tumor surgery and the aim of the method is to completely remove the tumor. However, it is emphasized that sometimes the pituitary tumor cannot be completely removed due to its deepness or different complications. It is stated that machine learning methods have been studied frequently in recent years and have promising results in the medical field. Therefore, they propose a model that calculates whether the tumor can be completely removed after surgery using MR images of 140 transsphenoidal surgery patients. They used deep learning network and Knosp classification method for the proposed model. They used the Knosp method to grade the tumor between 0-3. They achieved 91% success in deep learning and 81% in Knosp classification (Staartjes et al., 2018).

Yu Qian et al. used 796 MR images of 149 patients in their study. Since the CNN deep learning method, they used in the study would give better results with a large number of images, they increased the number of images to 8756 by performing data augmentation. Of the 149 patients, 84 were patients with pituitary tumors and 65 were in the control group. They randomly separated the dataset for training and testing at a ratio of 8:2. In the study, it is emphasized that the pituitary tumor is a relatively small organ and its size may vary from person to person. For these reasons, it is emphasized that it is very difficult to detect a pituitary tumor manually, and it depends on the experience and personal interpretation of the doctor. In the preprocessing, first of all, the region of the pituitary gland in the images of all patients was manually segmented in 180x140 size. After the MR images were preprocessed separately for each of the Coronal, Horizontal, Sagittal, T2-weighted images, Coronal and Horizontal angles of the T1 images, they were processed in the CNN architecture they created, and the success of comprehensive segmentation was examined. In this study, in which they labeled patients with pituitary tumors and patients in the control group, they achieved an average classification success of 91% in the classification of patients with and without tumors (Qian et al., 2020).

Our study is a retrospective study, and the dataset will be created using preoperative contrast enhanced T1-weighted MR images of female patients who underwent pituitary tumor surgery with the transsphenoidal method in Kahramanmaraş Sütçü İmam University Faculty of Medicine, Department of Neurosurgery. Necessary ethics committee permissions were obtained to create the data set. Figure 1 shows an exemplary MR image of a pituitary tumor. In the data set, MR images of 45 women diagnosed with pituitary tumors were used. There are 33 MR images, 11 in each sequence, and a total of 1485 MR images were used for 45 patients. The size of these images is 414x393 pixels.

Figure 1: MR image of pituitary tumor.

Preprocessing

The MR images in the dataset are in Digital Imaging and Communications in Medicine (DICOM) image format. The images were converted to Non-Invasive Fetal Trisomy (NIFTI) format and patient information on the image was removed. In addition, the 15-year-old specialist doctor removed the completely black part of the MR image and selected only the relevant area (ROI). The specialist has determined that the size of 414x393 is suitable for the images to be processed. An example image is shown in Figure 2.

Figure 2: An exemplary MR image.

Figure 3 shows an example of manual labeling made on a 2D slice of a randomly selected patient.

Figure 3: Manual labeling on a 2D MR image using the 3D Slicer program.

There may be noise in the MR images due to the high magnetic field used in MR imaging and the movement of the patient during the shooting. One of the common methods for removing these noises in biomedical images is the Gaussian filter (Afshari et al., 2017; Srinivasa Reddy and Jaya, 2021).

It can be said that the Gaussian smoothing operator is a 2-dimensional convolution operator used to blur images and remove noise (Wang et al., 2022). Equation 1 contains the general formula of the Gaussian filter. Here, “σ” denotes standard deviation, “x” and “y” denotes the values ​​on the axes.

In this study, the necessary literature search for noise reduction (Afshari et al., 2017; Srinivasa Reddy and Jaya, 2021; Wang et al., 2022) was made and the Gaussian filter was applied to MR images. In Figure 4, the results obtained by applying a Gaussian filter to the image of a patient in the data set for the “σ” value of 0.5 and 2. The “σ” value was preprocessed with the appropriate value of 2 by examining the literature (Afshariet al., 2017; Srinivasa Reddy and Jaya, 2021; Wang et al., 2022).

Figure 4: Images with Gaussian filter applied.

3D Slicer Program

The 3D Slicer program is a free download (https://www.slicer.org/) program that offers the opportunity to process medical images in every slice. In this project, the pituitary tumor will be manually segmented in all slices using program tools. It is very easy to operate in the program with a simple interface. It is possible to access training documents with a simple explanation on the https://www.slicer.org site. Figure 5 shows the interface image of the program (Egger, 2012).

Figure 5: An example image of 3D manual segmentation with the 3D Slicer program.

3D-Unet

U-Net architecture, Convolutional Neural Network (CNN) is a deep learning method with the same logic. U-Net deep learning architecture can achieve higher success in medical images because it performs pixel-based segmentation. While CNN needs a large number of images for training, this is not so important for U-Net architecture. For example; (Song et al. 2019) used 6 different medical images and 3 different medical images to look at the segmentation success in the U-Net architecture (Çiçek et al., 2018; Song et al., 2019).

Due to the location of the pituitary gland and the tumor tissue that may occur here, and since the transsphenoidal method, one of the pituitary tumor surgery methods, is likely to damage the surrounding tissues, 3D imaging will be useful for tumor detection and surgery. In this study, the MR images of 45 patients who underwent pituitary tumor surgery with the transsphenoidal method will be calculated using the 3D U-Net deep learning method to calculate the success of automatic segmentation of the tumor. If the architecture in Figure 6 is examined, each convolutional layer uses the "ReLu" activation function, while the "Fully Connected Layer", which is in the CNN deep learning method, is not included. The first part of the architecture works by increasing the size and the second part is working by decreasing the size (Çiçek et al., 2018; Song et al., 2019).

Application details

MR images were taken with General Electric brand, Optima MR 360 advance model, 1.5 Tesla MR device. Work was done in Python using Keras. For the training and validation of the network, a desktop computer with a 3.6GHz Intel(R) i7 CPU and GTX 1060TI graphics card and 16GB GPU memory was used. Table 2 shows the hyperparameters used for training.

Table 2: Hyperparameters

In the study, 90% of the data set was used for training and 10% for testing. Of the 45 images, 40 were used for training and 5 for testing. 5 pieces of data used for training were not included in the training in any way. Figure 6 shows the achievements of 3D automatic segmentation. The estimated pixel values ​​stabilized after the 800th epoch. In the calculated accuracy value, high success was achieved in the previous epochs. The Loss function, which optimizes the model's pixel classification with high success, is the 950th epoch for the validation data set (5 pieces of data), and the minimum value has been achieved.

Figure 6: 3D Unet architecture.

Table 3 shows the accuracy results of 3D automatic segmentation. The IoU score of the automatic segmentation performed for 5 pieces of data that was not subject to any training was quite high with 0.97. Although the number of pixels in the tumor area is less compared to the background, a high IoU score was obtained here as well.

Table 3: Scores

In our study, the pituitary tumor image of patient 3, obtained as a result of manual and automatic segmentation, is shown in Figure 7. Since there are 11 slices in each of the sagittal, axial and frontal images in each data, 11 slices are tried to be shown separately in Figure 7. The areas shaded in orange in each of the slices show the pixels that manual and automatic segmentation collectively identify. It is possible to interpret from the excess of orange scans that the pixels are correctly classified in each slice. The areas scanned in red on the slices show the pituitary tumor manually labeled by the specialist doctor. Green areas show pixels that are not manually labeled and are classified as pituitary tumors by 3D-Unet. When 11 slices are examined in detail, the number of misclassified pixels is quite low. A small number of pixels that cannot be manually labeled and accurately predicted by the model are visible in the 10th slice. A small number of pixels, which are not in manual labeling and are classified as pituitary tumors by the model, are slightly visible in each of the 11 slice images.

Figure 7: The result of patient No. 3 obtained by manual and automatic segmentation.

Hormonal balance is very important for people to lead a healthy life. Disruption of the hormonal balance can not only affect the relevant organ, but also indirectly affect all organs. Pituitary Gland regulates the release of hormones, one of the most important parts of a healthy life. One of the effects that disrupt the activities of the pituitary gland is tumor formation. Diseases such as Cushing's, acromegaly, and growth hormone deficiency may occur as a result of a pituitary tumor. In addition, the tumor may act as a pituitary in some cases and cause the release of hormones to a level that harms human health. It can be said that hormonal balance is more important in women due to conditions such as fertility. Therefore, more attention is paid to the hormone balance in women, and the hormone values ​​that may be related to the diagnosis of any disease gain importance. The Pituitary Gland, which is the manager of hormone release, becomes even more demanding for women.

The pituitary gland is located at the base of the brain, behind the bridge of the nose, in the bony cavity called the Sella Turcica (Turkish saddle). As can be seen in Figure 5, it can be said that the brain has the deepest place. Again, as can be seen in Figure 5, it is very close to the optic nerves. The pressure of the tumor to be formed on the optic nerves may also affect the visual function of the person. The operation to be performed to clean the tumor is quite difficult due to the location of the gland. There is a method called Transsphenoidal, which cleans the tumor by reaching the pituitary through the nose. Since this method is difficult and requires good expertise, it can be performed in a limited number of centers. It is a very difficult operation due to the location of the tumor. The proximity of the pituitary to the optic nerves makes it mandatory not to damage the nerves while the tumor is being cleaned. In addition, it will be very difficult to completely remove the tumor located in the deepest part of the brain.

Expert decision support systems are objective systems that aim to help the doctor for diagnosis and treatment by using computer software. Considerable progress has been made in the processing of biomedical images, especially with the deep learning methods becoming competent. Namely, 3D automatic classification and segmentation can be done on images. It is understood from here that detailed information about the location and shape of the tumor can be obtained by means of 3D automatic segmentation. If only 2D segmentation was done on a single slice, this information would not be available. The classification of pituitary tumors using the 3D-Unet network in the women we recommend is very valuable in this respect. In this way, the surgical operation can be planned in detail. In this study, research was carried out with computer software that can automatically 3D segment the pituitary tumors of female patients in whom hormonal activities are more important. The high semantic segmentation success of up to 98% objectively demonstrates the success of the proposed model.

Funding Statement

Funding not received for the study

Ethics Committee Certificate

Our study is a retrospective study and data were obtained from patients who applied to Kahramanmaras Sutcu Imam University Faculty of Medicine, Department of Neurosurgery. Ethics Committee approval was obtained from the Brain and Nerve Surgery institution to obtain the data (Session Number:2021/04, Protocol Number:07).

Thanks

For his contributions, Thanks to Prof. Dr. İdiris ALTUN.