Medical Diagnosis with Multimodal Image Fusion Techniques
Article Sidebar
Main Article Content
Abstract
Image Fusion is an effective approach utilized to draw out all the significant information from the source images, which supports experts in evaluation and quick decision making. Multi modal medical image fusion produces a composite fused image utilizing various sources to improve quality and extract complementary information. It is extremely challenging to gather every piece of information needed using just one imaging method. Therefore, images obtained from different modalities are fused Additional clinical information can be gleaned through the fusion of several types of medical image pairings. This study's main aim is to present a thorough review of medical image fusion techniques which also covers steps in fusion process, levels of fusion, various imaging modalities with their pros and cons, and the major scientific difficulties encountered in the area of medical image fusion. This paper also summarizes the quality assessments fusion metrics. The various approaches used by image fusion algorithms that are presently available in the literature are classified into four broad categories i) Spatial fusion methods ii) Multiscale Decomposition based methods iii) Neural Network based methods and iv) Fuzzy Logic based methods. the benefits and pitfalls of the existing literature are explored and Future insights are suggested. Moreover, this study is anticipated to create a solid platform for the development of better fusion techniques in medical applications.
Article Details
References
James AP, Dasarathy BV (2014) Medical image fusion: a survey of the state of the art. Inf Fusion 19:4–19.
https://doi.org/10.1016/j.inffus.2013.12.002
Tirupal, T.; Mohan, B. C.; Kumar, S. Srinivas, (2021) Multimodal Medical Image Fusion Techniques – A Review,Current Signal Transduction Therapy, Volume 16, Number 2, 2021, pp. 142-163(22). https://doi.org/10.2174/1574362415666200226103116
S. Li , X. Kang , L. Fang , J. Hu , H. Yin , Pixel-level image fusion: a survey of the state of the art, Information Fusion 33 (2017) 100–112 .
Y. Liu , X. Chen , Z. Wang , Z.J. Wang , R.K. Ward , X. Wang , Deep learning for pixel-level image fusion: recent advances and future prospects, Information Fusion 42 (2018) 158–173 .
MITA. https://www.medicalimaging.org/about-mita/medical-imaging-primer/.
NIBIB: https://www.nibib.nih.gov/science-education/science-topics/ultrasound
H.A. Mohammed, M.A. Hassan, The image registration techniques for medical imaging (MRI-CT), Am. J. Biomed. Eng. 6 (2) (2016) 53–58, doi:10.5923/j.ajbe.20160602.02
J. Du, W. Li, K. Lu, B. Xiao, An overview of multi-modal medical image fusion, Neurocomputing 215 (2016) 3–20, DOI:10.1016/j.neucom.2015.07.160
Falko Busse, Diagnostic Imaging in Advances in Health care Technology Care Shaping the Future of Medical( 2006), 15-34. https://doi.org/10.1007/1-4020-4384-8
Grova C, Jannin P, Biraben A, Buvat I, Benali H, Bernard AM, Scarabin JM, Gibaud B. Phys Med Biol. 2003 Dec 21;48(24):4023-43. doi: 10.1088/0031-9155/48/24/003.
Nosrati R, Wronski M, Tseng CL, Chung H, Pejovi?-Mili? A, Morton G, Stanisz GJ. Postimplant Dosimetry of Permanent Prostate Brachytherapy: Comparison of MRI-Only and CT-MRI Fusion-Based Workflows. Int J Radiat Oncol Biol Phys. 2020 Jan 1;106(1):206-215. doi:10.1016/j.ijrobp.2019.10.009
Mishra HOS, Bhatnagar S. , MRI and CT image fusion based on wavelet transform.,Int J Inf Comp Tech 2014; 4(1): 47-52.
Yang R, Lu H, Wang Y, Peng X, Mao C, Yi Z, Guo Y, Guo C. CT-MRI Image Fusion-Based Computer-Assisted Navigation Management of Communicative Tumors Involved the Infratemporal-Middle Cranial Fossa. J Neurol Surg B Skull Base. 2021 Jul;82(Suppl 3):e321-e329. doi: 10.1055/s-0040-1701603.
Monzen Y, Tamura A, Okazaki H, Kurose T, Kobayashi M, Kuraoka M. SPECT/CT Fusion in the Diagnosis of Hyperparathyroidism. Asia Ocean J Nucl Med Biol. 2015 Winter;3(1):61-5. PMID: 27408883; PMCID: PMC4937692.
Leiner Barba-J, Lorena Vargas-Quintero, Jose A. Calderón-Agudelo, Bone SPECT/CT image fusion based on the discrete Hermite transform and sparse representation, Biomedical Signal Processing and Control,Volume 71, Part A,2022,103096,ISSN 1746-8094. https://doi.org/10.1016/j.bspc.2021.103096.
Loeffelbein DJ, Souvatzoglou M, Wankerl V, Martinez-Möller A, Dinges J, Schwaiger M, Beer AJ. PET-MRI fusion in head-and-neck oncology: current status and implications for hybrid PET/MRI. J Oral Maxillofac Surg. 2012 Feb;70(2):473-83. doi: 10.1016/j.joms.2011.02.120.
Lin Q, Qi Q, Hou S, Chen Z, Jiang N, Zhang L, Lin C. Application of Pet-CT Fusion Deep Learning Imaging in Precise Radiotherapy of Thyroid Cancer. J Healthc Eng. 2021 Aug 5;2021:2456429. doi: 10.1155/2021/2456429.
Holupka EJ, Kaplan ID, Burdette EC, Svensson GK. Ultrasound Image fusion for external beam radiotherapy for prostate cancer. Int J Radiat Oncol Biol Phys 1996; 35(5): 975-84. http://dx.doi.org/10.1016/03603016(96)00231-3
Hosseini HG, Alizad A, Fatemi M. Integration of vibro acoustography imaging modality with the traditional mammography. Int J Biomed Imaging 2007; 2007: 40980.
Tirupal T., Mohan Chandra B. and Kumar Srinivas S., Multimodal Medical Image Fusion Techniques – A
Review, Current Signal Transduction Therapy, 2021, 16(2), https://dx.doi.org/10.2174/1574362415666200226103116
Yuri L-S, Vince DC, Tiilay A. Two Models for Fusion of Medical Imaging Data: Comparison and Connections. IEEE ICASSP 2017; pp. 6165-8.
Mahaboob BM, Tirupal T. On the Use of Spatial Frequency Technique for Detection of Brain Tumors in Medical Images. Int J Tech Research in Engineering 2015; 2(12): 3169-74.
H. G. Hosseini, A. Alizad, M. Fatemi, Fusion of vibro-acoustography images and X-ray mammography, in: 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2006, EMBS’06, IEEE, 2006, pp. 2803–2806.
Charles R. Hatt, Ameet K. Jain, Vijay Parthasarathy, Andrew Lang, Amish N. Raval, MRI—3D ultrasound—X-ray image fusion with electromagnetic tracking for transendocardial therapeutic injections: In-vitro validation and in-vivo feasibility,Computerized Medical Imaging and Graphics,Volume 37, Issue 2,2013,Pages 162-173,ISSN 0895-6111,https://doi.org/10.1016/j.compmedimag.2013.03.006.
Ghoshhajra BB, Takx RAP, Stone LL, Girard EE, Brilakis ES, Lombardi WL, Yeh RW, Jaffer FA. Real-time fusion of coronary CT angiography with x-ray fluoroscopy during chronic total occlusion PCI. Eur Radiol. 2017 Jun;27(6):2464-2473. doi: 10.1007/s00330-016-4599-5.
Hopp T, Baltzer P, Dietzel M, Kaiser WA, Ruiter NV. 2D/3D image fusion of X-ray mammograms with breast MRI: visualizing dynamic contrast enhancement in mammograms. Int J Comput Assist Radiol Surg. 2012 May;7(3):339-48. doi: 10.1007/s11548-011-0623-z.
NIH:https://www.nibib.nih.gov/science-education/science-topics/computed-tomography-ct
Liu SF, Lu J, Wang H, Han Y, Wang DF, Yang LL, et al. Computed tomography-magnetic resonance imaging fusion-guided iodine-125 seed implantation for single malignant brain tumor: Feasibility and safety. J Can Res Ther 2019;15:818-24.
Bavirisetti DP, Kollu V, Gang X, Dhuli R. Fusion of MRI and CT images using guided image filter and image statistics. Int. J. Imaging Syst. Technol. 2017;27:227–237. https://doi.org/10.1002/ima.22228
Hou, R., Zhou, D., Nie, R. et al. Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model. Med Biol Eng Comput 57, 887–900 (2019). https://doi.org/10.1007/s11517-018-1935-8.
Polinati, S.; Bavirisetti, D.P.;Rajesh, K.N.V.P.S.; Naik, G.R.; Dhuli,R. The Fusion of MRI and CT Medical Images Using Variational Mode Decomposition. Appl. Sci. 2021, 11,10975. https://doi.org/10.3390/app112210975
Rong Yang , Han Lu , Yang Wang , Xin Peng , Chi Mao , Zhiqiang Yi , Yuxing Guo , Chuanbin Guo,CT-MRI Image Fusion-Based Computer-Assisted Navigation Management of Communicative Tumors Involved the Infratemporal-Middle Cranial Fossa,J Neurol Surg B Skull Base 2021; 82(S 03): e321-e329,DOI: 10.1055/s-0040-1701603
Xiao Ning, Yang Wanting, Qiang Yan, Zhao Juanjuan, Hao Rui, Lian Jianhong, Li Shuo,PET and CT Image Fusion of Lung Cancer With Siamese Pyramid Fusion Network, Frontiers in Medicine,VOLUME=9,2022,2296-858X, DOI:10.3389/fmed.2022.792390.
S. Guruprasad, M. Z. Kurian and H. N. Suma, "Fusion of CT and PET Medical Images Using Hybrid Algorithm DWT-DCT-PCA," 2015 2nd International Conference on Information Science and Security (ICISS), 2015, pp. 1-5, doi: 10.1109/ICISSEC.2015.7371004.
Tao Zhou , Huiling Lu ,Fuyuan Hu ,Hongbin Shi,Shi Qiu ,Huiqun Wang,A New Robust Adaptive Fusion Method for Double-Modality Medical Image PET/CT, Artificial Intelligence for Medical Image Analysis, Volume 2021 |Article ID 8824395 | https://doi.org/10.1155/2021/8824395
Amini M, Nazari M, Shiri I, Hajianfar G, Deevband MR, Abdollahi H, Arabi H, Rahmim A, Zaidi H. Multi-level multi-modality (PET and CT) fusion radiomics: prognostic modeling for non-small cell lung carcinoma. Phys Med Biol. 2021 Oct 14;66(20). doi: 10.1088/1361-6560/ac287d.
Simanek M, Koranda P. SPECT/CT imaging in breast cancer - current status and challenges. Biomed Pap Med Fac Univ Palacky Olomouc Czech Repub. 2016 Dec;160(4):474-483. doi: 10.5507/bp.2016.036.
Kider, M. S. ., Qader, B. A. ., & Majeed, A. A. . (2023). Creating Online Tools for Theoretical Resource Management. International Journal of Intelligent Systems and Applications in Engineering, 11(1s), 193–200. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/2492
Monzen Y, Tamura A, Okazaki H, Kurose T, Kobayashi M, Kuraoka M. SPECT/CT Fusion in the Diagnosis of Hyperparathyroidism. Asia Ocean J Nucl Med Biol. 2015 Winter;3(1):61-5.
Z. Zhao, L. Li, F. Li, L. Zhao, Single photon emission computed tomography/ spiral computed tomography fusion imaging for the diagnosis of bone metastasis in patients with known cancer, Skel. Radiol. 39 (2) (2010) 147–153.
A. Hakime, T. de Baere, F. Deschamps, P. Rao, A. Auperin, E. Marques de Carvalho, Clinical evaluation of spatial accuracy of a fusion imaging technique combining previously-acquired computed tomography and real time ultrasound for imaging of liver tumors, J. Vasc. Interven. Radiol. 21 (2) (2010) S49.
European Society of Radiology (ESR). Abdominal applications of ultrasound fusion imaging technique: liver, kidney, and pancreas. Insights Imaging 10, 6 (2019). https://doi.org/10.1186/s13244-019-0692-z.
Musafargani, S., Ghosh, K.K., Mishra, S. et al. PET/MRI: a frontier in era of complementary hybrid imaging. European J Hybrid Imaging 2, 12 (2018). https://doi.org/10.1186/s41824-018-0030-6.
Umer Javed, Muhammad Mohsin Riaz, Abdul Ghafoor, Syed Sohaib Ali, Tanveer Ahmed Cheema, "MRI and PET Image Fusion Using Fuzzy Logic and Image Local Features", The ScientificWorldJournal, vol. 2014, ArticleID 708075, 8 pages, 2014. https://doi.org/10.1155/2014/708075
M.C. Dastjerdi, A. Karimian, H. Afarideh, A. Mohammadzadeh, FMDIB: a software tool for fusion of MRI and DHC-SPECT images of brain, in: World Congress on Medical Physics and Biomedical Engineering, September 7–12, 2009, Munich, Germany, Springer, 2009, pp. 741–744.
Hamid Reza Shahdoosti, Zahra Tabatabaei,MRI and PET/SPECT image fusion at feature level using ant colony based segmentation,Biomedical Signal Processing and Control,Volume 47,2019,Pages 63-74,ISSN 1746-8094,https://doi.org/10.1016/j.bspc.2018.08.017.
Kongnyuy M, George AK, Rastinehad AR, Pinto PA. Magnetic Resonance Imaging-Ultrasound Fusion-Guided Prostate Biopsy: Review of Technology, Techniques, and Outcomes. Curr Urol Rep. 2016 Apr; 17(4):32. doi: 10.1007/s11934-016-0589-z.
Paparo, F., Piccazzo, R., Cevasco, L. et al. Advantages of percutaneous abdominal biopsy under PET-CT/ultrasound fusion imaging guidance: a pictorial essay. Abdom Imaging 39, 1102–1113 (2014). https://doi.org/10.1007/s00261-014-0143-8
Freesmeyer, M., Winkens, T. (2016). Ultrasound Fusion (SPECT/US). In: Herrmann, K., Nieweg, O., Povoski, S. (eds) Radioguided Surgery. Springer, Cham. https://doi.org/10.1007/978-3-319-26051-8_28
Y. Yang, Y. Que, S. Huang, P. Lin, Multimodal sensor medical image fusion based on type-2 fuzzy logic in NSCT domain, IEEE Sensor. J. 16 (10) (2016) 3735–3745, https://doi.org/10.1109/JSEN.2016.2533864.
B. Meher, S. Agrawal, R. Panda, A. Abraham, A survey on region based image fusion methods, Inf. Fusion 48 (2019) 119–132, July 2018.https://doi.org/10.1016/j. inffus.2018.07.010.
Bing Huang, Feng Yang, Mengxiao Yin, Xiaoying Mo, Cheng Zhong, "A Review of Multimodal Medical Image Fusion Techniques", Computational and Mathematical Methods in Medicine, vol. 2020, Article ID 8279342, 16 pages, 2020. https://doi.org/10.1155/2020/8279342
F.E.Z.A. El-Gamal, M. Elmogy, A. Atwan, Current trends in medical image registration and fusion, Egypt. Inform. J. 17 (1) (2016) 99–124, https://doi.org/ 10.1016/j.eij.2015.09.002.
M. R. Metwalli, A. H. Nasr, O. S. Farag Allah and S. El-Rabaie, "Image fusion based on principal component analysis and high-pass filter," 2009 International Conference on Computer Engineering & Systems, 2009, pp. 63-70, doi: 10.1109/ICCES.2009.5383308.
Keyur N. Brahmbhatt,Ramji M. Makwana, comparative study on image fusion methods in spatial domain, International Journal of Advanced Research in Engineering and Technology (IJARET), 2013, 4(2), PP161-166
C. -I. Chen, "Fusion of PET and MR Brain Images Based on IHS and Log-Gabor Transforms," in IEEE Sensors Journal, vol. 17, no. 21, pp. 6995-7010, 1 Nov.1, 2017, doi: 10.1109/JSEN.2017.2747220.
Krishn, A., Bhateja, V., Himanshi, Sahu, A. (2015). PCA Based Medical Image Fusion in Ridgelet Domain. In: Satapathy, S., Biswal, B., Udgata, S., Mandal, J. (eds) Proceedings of the 3rd International Conference on Frontiers of Intelligent Computing: Theory and Applications (FICTA) 2014. Advances in Intelligent Systems and Computing, vol 328. Springer, Cham. https://doi.org/10.1007/978-3-319-12012-6_52
S. Madanala and K. J. Rani, "PCA-DWT based medical image fusion using non sub-sampled contourlet transform," 2016 International Conference on Signal Processing, Communication, Power and Embedded System (SCOPES), 2016, pp. 1089-1094, doi: 10.1109/SCOPES.2016.7955608.
H. Yan and Z. Li, "A Multi-modal Medical Image Fusion Method in Spatial Domain," 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), 2019, pp. 597-601, doi: 10.1109/ITNEC.2019.8729143.
Q. Guihong, Z. Dali, Y. Pingfan, Medical image fusion by wavelet transform modulus maxima, Opt Express 9 (4) (2001) 184, https://doi.org/10.1364/ oe.9.000184.
Ramasamy, Asokan & Chinnathambi, Kalaiselvi. (2016). Medical image fusion using stationary wavelet transform with different wavelet families. Pakistan Journal of Biotechnology. 13. 10-14.
Nikolov, S., Hill, P., Bull, D., Canagarajah, N. (2001). Wavelets for Image Fusion. In: Petrosian, A.A., Meyer, F.G. (eds) Wavelets in Signal and Image Analysis. Computational Imaging and Vision, vol 19. Springer, Dordrecht. https://doi.org/10.1007/978-94-015-9715-9_8.
Xianghai Wang, Yutong Shen, Zhiguang Zhou and Lingling Fang (2015) Journal of Optics, Volume 17, Number 5 , DOI 10.1088/2040-8978/17/5/055702.
Xia J, Chen Y, Chen A, Chen Y. Medical Image Fusion Based on Sparse Representation and PCNN in NSCT Domain. Comput Math Methods Med. 2018 May 24;2018:2806047. doi: 10.1155/2018/2806047. PMID: 29991960; PMCID: PMC5994313
Gai, Di & Xuanjing, Shen & Cheng, Hang & Chen, Haipeng. (2019). Medical Image Fusion via PCNN Based on Edge Preservation and Improved Sparse Representation in NSST Domain. IEEE Access. PP. 1-1. 10.1109/ACCESS.2019.2925424.
C. Bhuvaneswari, P. Aruna, D. LoganathanA new fusion model for classification of the lung diseases using genetic algorithm,Egyptian Informatics Journal,Volume 15, Issue 2,2014,Pages 69-77,ISSN 1110-8665, https://doi.org/10.1016/j.eij.2014.05.001.
E. Daniel, "Optimum Wavelet-Based Homomorphic Medical Image Fusion Using Hybrid Genetic–Grey Wolf Optimization Algorithm," in IEEE Sensors Journal, vol. 18, no. 16, pp. 6804-6811, 15 Aug.15, 2018, doi: 10.1109/JSEN.2018.2822712.
Kavitha, S., Bharathi, B.,Sasikala, P., Chandraleka, D. and Ashwini, V. (2018) ‘Modified dual channel PCNN algorithm with hybrid edge enhancement approach for multimodality brain image fusion’, Int. J. Biomedical Engineering and Technology, Vol. 28,No. 2, pp.120–146.
Manchanda M, Sharma R (2016) A novel method of multimodal medical image fusion using fuzzy
transform. Journal of Vis Commun Image Represent 40:197–217. https://doi.org/10.1016/j.jvcir.2016.06.021
.Y. Yang, Y. Que, S. Huang and P. Lin, "Multimodal Sensor Medical Image Fusion Based on Type-2 Fuzzy Logic in NSCT Domain," in IEEE Sensors Journal, vol. 16, no. 10, pp. 3735-3745, May15, 2016, doi: 10.1109/JSEN.2016.2533864.
Verma, D. ., Reddy, A. ., & Thota, D. S. . (2021). Fungal and Bacteria Disease Detection Using Feature Extraction with Classification Based on Deep Learning Architectures. Research Journal of Computer Systems and Engineering, 2(2), 27:32. Retrieved from https://technicaljournals.org/RJCSE/index.php/journal/article/view/29
Andrew Hernandez, Stephen Wright, Yosef Ben-David, Rodrigo Costa, David Botha. Risk Assessment and Management with Machine Learning in Decision Science. Kuwait Journal of Machine Learning, 2(3). Retrieved from http://kuwaitjournals.com/index.php/kjml/article/view/196
Tirupal T, Mohan BC, Kumar SS. Multimodal medical image fusion based on Sugeno’s intuitionistic fuzzy sets. ETRI J 2017; 39(2): 173-80. https://doi.org/10.4218/etrij.17.0116.0568.
Kaur H, Kumar S, Behgal KS, Sharma Y. Multi-Modality Medical Image Fusion Using Cross-Bilateral Filter and Neuro-Fuzzy Approach. J Med Phys. 2021 Oct-Dec;46(4):263-277. doi: 10.4103/jmp.JMP_14_21. Epub 2021 Dec 2. PMID: 35261496; PMCID: PMC8853451.
N. Nagaraja Kumar, T. Jayachandra Prasad and K. Satya Prasad, "Multimodal Medical Image Fusion with Improved Multi-Objective Meta-Heuristic Algorithm with Fuzzy Entropy", Journal of Information & Knowledge Management,https://doi.org/10.1142/S0219649222500630.
Sakura Nakamura, Machine Learning in Environmental Monitoring and Pollution Control , Machine Learning Applications Conference Proceedings, Vol 3 2023.
P. Jagalingam, Arkal Vittal Hegde “A Review of Quality Metrics for Fused Image”,Aquatic Procedia,Volume 4,2015,Pages 133-142,ISSN 2214-241X, https://doi.org/10.1016/j.aqpro.2015.02.019.