Several extensions to the considered architectures are possible in oder to solve the follow-ing problems: abscence of significant amount of the data samples, independent patchwise processing and development of an architecture which could be used for segmentation of the blood vessels, optic disc, macula and lesions.
Collecting the data from the patients is a difficult, expensive and time consuming process which requires proper imaging setup and educated medical staff. Thus, it would be ben-eficial to develop a generative model which can be used to produce more data from the existing dataset taking into account provided ground truth data.
Recently a lot of papers about adversarial architectures have been published [61] which typically consist of a generator and a discriminator. The generator tries to learn the data distribution, whereas the discriminator is used to estimate the probability that the sample came from the training set and not from the generator. The particular interesting type of the architecture is a conditional generative adversarial network [62], where both the generator and discriminator can be conditioned on the ground truth data, e.g., label maps with lesions, blood vessels, optic disc and macula.
Another approach to deep generative modeling is based on the generative version of AEs namely variational autoencoders [63] which learns conditional distributions of the data given its hidden representation and the hidden representation given the data. Variational autoencoders also have the conditional version [64] which gives all the advantages of semi-supervised learning and cross-modality learning. Since conditional variational au-toencoders can be also described in terms of encoders and decoders, it can be naturally included in the architectures considered in this thesis.
The segmentation of large images requires a huge amount of computational resources, and the simplest solution is a patchwise processing which may introduce checkerboard arte-facts in the segmentation results. Another problem arises from the independent processing of the patches: the segmentation result obtained in adjacent patches might be included as prior information for the current patch segmentation. This problem can be solved using modern deep recurrent visual attention mechanisms [65]. It has been also shown that visual attention mechanisms can be utilized in order to better analyze the objects with different scales [66], and, consequently, it may help to build one architecture which can be used to segment all interesting objects in the images including low scale lesions.
All considered improvements may help to build more powerful deep learning systems and to solve all the mentioned problem, but in case of DiaRetDB2 dataset the main problem is the ground truth data for the vessels. In order to make further research more productive and less misleading, the vessels markings should be refined.
7 CONCLUSION
In this work, the four different architectures for retinal blood vessel segmentation have been implemented and tested. The considered architectures are SegNet and three adap-tations of SegNet with dimensionality reduction layers. It was shown that the utilization of the dimensionality reduction layers did not lead to any significant improvements in the performance, and it increases the amount of training time.
The comparison of segmentation results for the RGB and hyperspectral images is given.
The utilization of the spectral provided minor improvements in the blood vessel segmenta-tion results compare to the RGB images. But in the case of RGB images, both the training and inference can be performed faster.
The experiments with MC dropout and the uncertainty estimation were carried out. MC dropout moderately improved the performance of the networks when it was used with DR layers, and it allows to estimate the uncertainty of the activations. The produced uncer-tainty maps were similar to the images representing missclassified pixels.
REFERENCES
[1] Jogi,Basic Ophthalmology. New Delhi, India: Jaypee Brothers Medical Publishers, 2008.
[2] A. ElTanboly, M. Ismail, A. Shalaby, A. Switala, A. El-Bazy, S. Schaal, G. Gimel’farb, and M. El-Azab, “A computer aided diagnostic system for detect-ing diabetic retinopathy in optical coherence tomography images,”Medical Physics, vol. 10, no. 3, pp. 182–188, 2016.
[3] T. Kauppi,Eye fundus image analysis for automatic detection of diabetic retinopathy.
PhD thesis, Lappeenranta University of Technology, 2010.
[4] L. Laaksonen, Spectral retinal image processing and analysis for ophthalmology.
PhD thesis, Lappeenranta University of Technology, 2016.
[5] M. D. Abràmoff, M. K. Garvin, and M. Sonka, “Retinal imaging and image analysis,”
IEEE reviews in biomedical engineering, vol. 3, pp. 169–208, 2010.
[6] C. R. Baumal, “Clinical applications of optical coherence tomography,” Current opinion in ophthalmology, vol. 10, no. 3, pp. 182–188, 1999.
[7] J. Duker, N. Waheed, and D. Goldman,Handbook of Retinal OCT: Optical Coher-ence Tomography, 1e. Saunders, 1st ed., 2014.
[8] I. Gurov and M. Volynsky, “Interference fringe analysis based on recurrence compu-tational algorithms,”Optics and Lasers in Engineering, vol. 50, pp. 514–521, 2012.
[9] “Is OCT Worth It?.”https://www.photonics.com/Article.aspx?AID=36339.
Accessed: 2017-01-14.
[10] “New & Used Optical Coherence Tomography (OCT) | Buy Used Op-tical Coherence Tomography (OCT) Equipment.” http://www.medwow.com/
used-optical-coherence-tomography-oct-equipment/481.med. Accessed:
2017-01-14.
[11] L. Wang and C. Zhao,Hyperspectral Image Processing. Springer, 2015.
[12] C. Sinthanayothin, J. F. Boyce, T. H. Williamson, H. L. Cook, E. Mensah, S. Lal, and D. Usher, “Automated detection of diabetic retinopathy on digital fundus images,”
Diabetic Medicine, vol. 19, no. 2, pp. 105–112, 2002.
[13] G. Gardner, D. Keating, T. Williamson, and A. Elliott, “Automatic detection of di-abetic retinopathy using an artificial neural network: a screening tool.,”The British Journal of Ophthalmology, vol. 80, pp. 940–944, 1996.
[14] K. Malathi and R. Nedunchelian, “Detecting and classifying diabetic retinopathy in fundus retina images using artificial neural networks-based firefly clustering al-gorithm,” ARPN Journal of Engineering and Applied Sciences, vol. 11, no. 5, pp. 3419–3426, 2016.
[15] Y. Bengio, “Learning deep architectures for AI,”Foundations and trends in Machine Learning, vol. 2, no. 1, pp. 1–127, 2009.
[16] S. Srinivas, R. K. Sarvadevabhatla, K. R. Mopuri, N. Prabhu, S. S. Kruthiventi, and R. V. Babu, “A taxonomy of deep convolutional neural nets for computer vision,”
arXiv preprint arXiv:1601.06615, 2016.
[17] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A.
van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,”arXiv preprint arXiv:1702.05747, 2017.
[18] S. Wang, Y. Yin, G. Cao, B. Wei, Y. Zheng, and G. Yang, “Hierarchical retinal blood vessel segmentation based on feature and ensemble learning,” Neurocomput-ing, vol. 149, pp. 708–717, 2015.
[19] M. v. Grinsven, B. v. Ginneken, C. B. Hoyng, T. Theelen, and C. I. Sanchez, “Fast convolutional neural network training using selective data sampling: Application to haemorrhage detection in color fundus images,” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1273 – 1284, 2016.
[20] P. Liu, H. Zhang, and K. B. Eom, “Active Deep Learning for Classification of Hy-perspectral Images,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 99, 2016.
[21] W. Zhao and S. Du, “Spectral;Spatial Feature Extraction for Hyperspectral Image Classification: A Dimension Reduction and Deep Learning Approach,”IEEE Trans-actions on Geoscience and Remote Sensing, vol. 54, no. 8, 2016.
[22] A. Romero, C. Gatta, and G. Camps-Valls, “Unsupervised Deep Feature Extraction for Remote Sensing Image Classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 3, 2016.
[23] X. Ma, J. Geng, and H. Wang, “Hyperspectral image classification via contextual deep learning,”EURASIP Journal on Image and Video Processing, vol. 2015, no. 1, pp. 20 – 28, 2015.
[24] Y. Chen, X. Zhao, and X. Jia, “Spectral-Spatial Classification of Hyperspectral Data Based on Deep Belief Network,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 8, no. 6, 2015.
[25] “Figure 3 - Normal Fundus Image.” http://www.rpfightingblindness.org.
uk/index.php?pageid=201&tln=aboutrp. Accessed: 2017-01-14.
[26] P. Fält, J. Hiltunen, M. Hauta-Kasari, I. Sorri, V. Kalesnykiene, J. Pietilä, and H. Uusitalo, “Spectral Imaging of the Human Retina and Computationally Deter-mined Optimal Illuminants for Diabetic Retinopathy Lesion Detection,”Journal of Imaging Science and Technology, vol. 55, no. 3, pp. 253–263, 2011.
[27] Y. Zheng, D. Stambolian, J. O’Brien, and J. C. Gee, “Optic disc and cup segmentation from color fundus photograph using graph cut with priors,” inInternational Confer-ence on Medical Image Computing and Computer-Assisted Intervention, pp. 75–82, Springer, 2013.
[28] G. Mahendran, R. Dhanasekaran, and N. Devi, “Morphological process based seg-mentation for the detection of exudates from the retinal images of diabetic patients,”
inIEEE International Conference on Advanced Communications, Control and Com-puting Technologies, pp. 1466–1470, 2014.
[29] A. S. Potapov, “Principle of representational minimum description length in image analysis and pattern recognition,”Pattern Recognition and Image Analysis, vol. 22, no. 1, pp. 82–91, 2012.
[30] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,”Neural Networks, vol. 2, no. 5, pp. 359–366, 1989.
[31] S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah, “Comparative study of deep learning software frameworks,”arXiv preprint arXiv:1511.06435, 2015.
[32] I. Goodfellow, Y. Bengio, and A. Courville,Deep Learning. MIT Press, 2016.
[33] A. K. Kolmogorov, “On the representation of continuous functions of several vari-ables by superposition of continuous functions of one variable and addition,” Dok-lady Akademii Nauk SSSR, vol. 114, pp. 369–373, 1957.
[34] G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathemat-ics of control, signals and systems, vol. 2, no. 4, pp. 303–314, 1989.
[35] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,”Neural computation, vol. 18, no. 7, pp. 1527–1554, 2006.
[36] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”arXiv preprint arXiv:1502.03167, 2015.
[37] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,
“Dropout: a simple way to prevent neural networks from overfitting.,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
[38] Y. Gal,Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
[39] D. P. Kingma, T. Salimans, and M. Welling, “Variational dropout and the local reparameterization trick,” in Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, eds.), pp. 2575–2583, Curran Associates, Inc., 2015.
[40] J. Rahebi and F. Hardalaç, “Retinal Blood Vessel Segmentation with Neural Network by Using Gray-Level Co-Occurrence Matrix-Based Features,” Journal of Medical Systems, vol. 38, no. 8, p. 85, 2014.
[41] M. García, M. I. López, D. Álvarez, and R. Hornero, “Assessment of four neural net-work based classifiers to automatically detect red lesions in retinal images,”Medical Engineering and Physics, vol. 32, no. 10, pp. 1085–1093, 2010.
[42] A. Osareh, M. Mirmehdi, B. Thomas, and R. Markham, “Automated identification of diabetic retinal exudates in digital colour images,”British Journal of Ophthalmology, vol. 87, no. 10, pp. 1220–1223, 2003.
[43] M. Nandy and M. Banerjee, “Retinal vessel segmentation using Gabor filter and arti-ficial neural network,” in Third International Conference on Emerging Applications of Information Technology (EAIT), 2012, pp. 157–160, 2012.
[44] S. W. Franklin and S. E. Rajan, “Retinal vessel segmentation employing ANN tech-nique by Gabor and moment invariants-based features,” Applied Soft Computing, vol. 22, pp. 94–100, 2014.
[45] M. Ceylan and H. Yaçar, “Blood vessel extraction from retinal images using Com-plex Wavelet Transform and ComCom-plex-Valued Artificial Neural Network,” in 2013 36th International Conference on Telecommunications and Signal Processing (TSP), pp. 822–825, 2013.
[46] J. Staal, M. Abramoff, M. Niemeijer, M. Viergever, and B. van Ginneken, “Ridge based vessel segmentation in color images of the retina,”IEEE Transactions on Med-ical Imaging, vol. 23, no. 4, pp. 501–509, 2004.
[47] A. Hoover, V. Kouznetsova, and M. Goldbaum, “Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response,”IEEE Transac-tions on Medical imaging, vol. 19, no. 3, pp. 203–210, 2000.
[48] “Kaggle.” https://www.kaggle.com/c/diabetic-retinopathy-detection.
Accessed: 2017-01-14.
[49] Q. Li, B. Feng, L. Xie, P. Liang, H. Zhang, and T. Wang, “A Cross-Modality Learning Approach for Vessel Segmentation in Retinal Images,”IEEE transactions on medical imaging, vol. 35, no. 1, pp. 109–118, 2016.
[50] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A deep convo-lutional encoder-decoder architecture for image segmentation,” arXiv preprint arXiv:1511.00561, 2015.
[51] S. Yu, S. Jia, and C. Xu, “Convolutional neural networks for hyperspectral image classification,”Neurocomputing, vol. 219, pp. 88–98, 2017.
[52] S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz, T. Greer, B. ter Haar Romeny, J. B. Zimmerman, and K. Zuiderveld, “Adaptive histogram equalization and its variations,”Computer vision, graphics, and image processing, vol. 39, no. 3, pp. 355–368, 1987.
[53] S. Ruder, “An overview of gradient descent optimization algorithms,”arXiv preprint arXiv:1609.04747, 2016.
[54] M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXiv preprint arXiv:1212.5701, 2012.
[55] P. Charles, “Project title.” https://github.com/charlespwd/project-title, 2013.
[56] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Is-ard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015. Software available from tensorflow.org.
[57] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cuDNN: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014.
[58] Itseez, “Open source computer vision library.” https://github.com/itseez/
opencv, 2015.
[59] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon-del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Courna-peau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,”Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
[60] J. D. Hunter, “Matplotlib: A 2D graphics environment,” Computing In Science &
Engineering, vol. 9, no. 3, pp. 90–95, 2007.
[61] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” inAdvances in Neural Information Processing Systems, pp. 2672–2680, 2014.
[62] M. Mirza and S. Osindero, “Conditional generative adversarial nets,”arXiv preprint arXiv:1411.1784, 2014.
[63] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” arXiv preprint arXiv:1312.6114, 2013.
[64] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional generative models,” inAdvances in Neural Information Processing Sys-tems 28(C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, eds.), pp. 3483–3491, Curran Associates, Inc., 2015.
[65] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, “Recurrent models of visual attention,” inAdvances in Neural Information Processing Systems, pp. 2204–2212, 2014.
[66] L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention to scale: Scale-aware semantic image segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3640–3649, 2016.