5.5 Experimental results
5.5.4 Computation time
In study I, the computation time for the segmentation of one high-resolution SBEM dataset of 1000×1000×280 voxels on a 4-core Intel CPU 3.41 computer with 64 RAM using MATLAB R2017b was approximately 24 h, plus 5-7 h of BM4D filtering of the volumes.
In study II, DeepACSON required approximately five days to segment one low-resolution SBEM dataset of 4000×2000×1200 voxels into its final segmentation.
BM4D filtering consumed 40% of the DeepACSON computation time. We ran BM4D filtering on non-overlapping patches of the SBEM volumes to enable parallel processing. The number of floating-point operations required by BM4D was 𝑂(𝑁) with large constants, where 𝑁 is the number of voxels. The CSD algorithm
consumed approximately 30% of the DeepACSON computation time. The time complexity of skeletonization was 𝑂(𝑛 𝑁Ωlog𝑁Ω), where 𝑛 is the number of skeleton branches, and 𝑁Ω is the number of voxels of a discrete object. The 𝑁Ωlog𝑁Ω factor was obtained from the fast marching algorithm (Sethian, 1996). The time
complexity to determine a critical point was 𝑂(𝑁𝑝), where 𝑁𝑝 represented the number of inquiry points to check for the cross-sectional changes in a
decomposition interval. We calculated the overall time complexity of the CSD algorithm as 𝑂(𝑛 𝑁Ωlog𝑁Ω) + 𝑂(𝑁𝑝). The inference time of the FCNs corresponded to approximately 10% of the DeepACSON computation time. In the general
analysis of the time complexity of FCNs, we refer to (He and Sun, 2015). Comparing the computation time of DeepACSON, DeepEM2D/3D, and FFN techniques on a computer with an NVIDIA Tesla V100-32 GB GPU, 2×Intel Xeon E5 2630 CPU 2.4 GHz, and 512 GB RAM showed that DeepEM2D/3D had the shortest computation time (about 1 minute) as the segmentation essentially relied on an Inception-ResNet-v2 network and watershed segmentation. DeepACSON required about 4 minutes (using 15 CPU cores) to segment the test datasets. FFN required the longest computation time of about 28 minutes for an end-to-end segmentation.
6 DISCUSSION
This thesis developed two segmentation pipelines, ACSON and DeepACSON that traced ultrastructures in the EM volumes of white matter, and a shape
decomposition algorithm, CSD, used as the instance segmentation of the DeepACSON pipeline. We developed ACSON based on an SRG algorithm that automatically segmented white matter into myelin, myelinated axons,
mitochondria, and cell bodies/processes in small field-of-view high-resolution SBEM datasets and extracted the morphology of myelinated axons. We developed DeepACSON on a CNN-based semantic segmentation and shape decomposition-based instance segmentation. DeepACSON was trained by the ACSON
segmentation of high-resolution SBEM volumes, i.e., the training set was generated automatically. We applied DeepACSON on large field-of-view low-resolution SBEM volumes of white matter to segment long myelinated axons.
DeepACSON segmented hundreds of thousands of myelinated axons, thousands of cell nuclei, and millions of mitochondria in ten SBEM datasets with excellent evaluation scores. We also developed the CSD algorithm to decompose tubular structures into their semantic components to be applied as the DeepACSON instance segmentation step.
In study I, the SRG-based segmentation of ACSON combined with an edge detection technique allowed for robust segmentation of myelinated axons in high-resolution EM volumes. ACSON segmentation required the tuning of several parameters, such as a similarity threshold or a threshold for the volume of an ultrastructure, which were easy to set. Evaluations of ACSON demonstrated a substantial agreement between automated and manual segmentation of
myelinated axons. The accuracy of ACSON segmentation of myelinated axons (the weighted Dice coefficient was approximately 0.87) is comparable to the study of Lee et al. (Lee et al., 2019), where the authors reported a Dice coefficient of 0.92 on the segmentation of approximately a hundred selected myelinated axons.
Assessing the ACSON segmentation of unmyelinated axons, the weighted Dice coefficient was approximately 0.5, indicated frequent mismatches between automated and manual segmentation of unmyelinated axons because of faintly resolved membranes.
In study II, the top-down design of DeepACSON instance segmentation allowed the inclusion of a-priori knowledge, i.e., tubularity of myelinated axons, in the segmentation process, which made our method different from the bottom-up
design of the current segmentation techniques (Januszewski et al., 2018; Funke et al., 2019). In MaskExtend, Meirovitch et al. (Meirovitch et al., 2016) proposed a similar approach, detecting X-shape objects to find under-segmentation errors.
Unlike MaskExtend, which was limited to X-shaped objects, we allowed more general shapes for under-segmented objects. Evaluating DeepACSON showed that our method outperformed state-of-the-art techniques in segmenting
low-resolution EM volumes. DeepACSON had an equally robust performance in both sham-operated and TBI datasets, where abnormality in the shape of myelinated axons was more frequent in TBI. We also noticed that recall was lower than DeepACSON precision, i.e., a greater number of false negatives than false
positives. We speculate that discarding small over-segmented components of very thin myelinated axons before the shape decomposition step increased the
number of false negatives. Note that low-resolution imaging of an axon with a diameter smaller than 0.2 µm resulted in 13 cross-sectional voxels, which was difficult to trace even for an expert; Mikula et al. (Mikula, Binding and Denk, 2012) reported that an expert made an error in every 80 nodes of Ranvier in the white matter when the axonal diameter was greater than 0.5 µm and the resolution was 50×50×50 nm3.
In study III, we segmented hundreds of thousands of myelinated axons by the CSD parallelizability on high-performance computing servers. We compared the CSD algorithm to ACD and GCD, showing that CSD outperformed these techniques in decomposing voxel-based objects and was highly robust to surface noise when decomposing tubular objects. We remark that when the surface protrusion of the object was very irregular, the skeletonization could over-estimate the number of skeleton branches, causing CSD to over-segment the surface protrusion.
We proposed the adoption of low-resolution EM imaging to scan large fields of view of the white matter tissue in study II. We recommend low-resolution EM imaging to prevent time-consuming and expensive image acquisition schemes at high resolutions because, despite increases in the image acquisition rate (Wanner, Kirschmann and Genoud, 2015; Zheng et al., 2018; Maniates-Selvin et al., 2020), microscopes should continuously run for several months in order to obtain a cubic millimeter at very high resolutions, e.g., 4×4×40 nm3 (Maniates-Selvin et al., 2020).
On the other hand, imaging at low-resolutions, 50×50×50 nm3, reduces the imaging time and image size by approximately 200 fold. And yet, low-resolution imaging allows for axonal tracking, morphometry of myelinated axons, and spatial distribution analyses of myelinated axons, cell nuclei, and mitochondria.
Large field-of-view imaging makes it possible to quantify parameters whose measurement in a small field-of-view may reflect very local characteristics of the underlying ultrastructure. These parameters include the tortuosity of myelinated axons, inter-mitochondrial distance, and cell density. In addition, despite lowering the resolution to acquire large field-of-view SBEM volumes, we have still been able to measure axonal diameter and eccentricity with the same accuracy as in high-resolution datasets. For example, we measured an average axonal diameter for myelinated axons in the contralateral corpus callosum of sham #25 (sham #49) equal to 0.41 µm (0.44 µm) and 0.44 µm (0.50 µm) in the low- and high-resolution imaging, respectively. In addition, we measured an average eccentricity for myelinated axons in the contralateral corpus callosum of sham #25 (sham #49) equal to 0.71 (0.69) and 0.72 (0.71) in the low- and high-resolution imaging, respectively.
Previous white matter quantification studies were restricted to 2D
morphometry, simplifying the assumptions about axonal morphology. In this thesis, we presented an extensive 3D morphological analysis of SBEM volumes, demonstrating a substantial variation in the diameter of myelinated axons and that the cross-sections of myelinated axons were more likely to be elliptical than circular. The ACSON and DeepACSON morphological analyses of myelinated axons in sham-operated animals were in line with those described in a previous study (Kim and Juraska, 1997) measuring axon diameter in the rat corpus callosum.
In studies I and II, we quantified morphological changes in myelinated axons in white matter five months after a severe TBI. The white matter pathology after TBI is extremely complex. The initial response of axons to brain injury can be either degeneration or regeneration (Mierzwa et al., 2015; Armstrong et al., 2016).
Moreover, in humans, morphological alterations in axons can persist for years after an injury (Chen et al., 2009; Johnson, Stewart and Smith, 2013) and up to one year in rats (Rodriguez-Paez, Brunschwig and Bramlett, 2005). In studies I and II, we found that the axonal diameter of myelinated axons in the ipsilateral corpus callosum, ipsilateral cingulum, and contralateral cingulum was significantly smaller in TBI rats. Moreover, the density of myelinated axons was significantly lower in TBI rats, which may reflect the axonal degeneration occurring after the injury (Johnson, Stewart and Smith, 2013). We also found that TBI increased the
tortuosity of myelinated axons in the ipsilateral cingulum, which might reflect the prolonged damage in microtubules (Tang-Schomer et al., 2010). We subjectively observed that myelin delamination was more frequent in TBI animals, indicating that active myelin processes were still ongoing five months after the injury. We
also observed myelin delamination in sham-operated rats, which may be part of the natural dynamics of healthy myelin. It is worth noting that pockets in the myelin sheaths appeared more frequently in the injured animals. The
delamination of the myelin sheaths in healthy and injured brains at this chronic time point requires further investigation.
Ultrastructural tissue modeling aims to bridge the gap between macroscopic and cellular and subcellular tissue levels. Traditional tissue models have relied on simplified representations and assumptions of ultrastructural properties, such as assuming axons to be perfect cylinders or ignoring the variance in axonal diameter along the length of an axon (Alexander et al., 2010; Kamiya et al., 2017). We can replace simple biophysical models with more realistic models by segmenting brain tissue ultrastructures in 3D EM datasets. Such realistic 3D tissue models enable researchers to look into the causes of contrast in diffusion magnetic resonance imaging and its macroscopic changes in brain diseases (Salo et al., 2018, 2021;
Ginsburger et al., 2019; Lee et al., 2019; Palombo, Alexander and Zhang, 2019) or investigate conduction velocity in myelinated and unmyelinated axons
encountered in electrophysiology (Chomiak and Hu, 2009; Kwong et al., 2019).
The conducted studies have some limitations that can be addressed in future research. In ACSON and DeepACSON studies, the instance segmentation of myelin, i.e., assigning myelin to each intra-axonal space, was not addressed. In addition, we mainly focused on the segmentation and morphology analysis of myelinated axons; other structures such as unmyelinated axons are also important and should be studied in further work to extend the current analysis. We remark that the tissue shrinkage caused by fixation, staining, and sectioning can affect the quantification of axonal diameter and volumes when characterizing ultrastructural morphology (Virtanen et al., 1984). Although we obtained our SBEM datasets consistently from a specific location in the brain, in both sham-operated and TBI animals and from both hemispheres, the locations may still differ due to the small tissue size. Therefore, studies including more SBEM datasets from more subjects and/or locations in the brain and larger sample sizes will be necessary to improve quantitative ultrastructural determinations.
7 SUMMARY AND CONCLUSIONS
In this thesis, we developed automated algorithms to segment white matter ultrastructures in 3D EM volumes. We developed an SRG-based ACSON pipeline to segment white matter ultrastructures in the high-resolution small field-of-view SBEM datasets. DeepACSON was developed on a CNN-based semantic
segmentation and a shape decomposition-based instance segmentation. We used ACSON segmentation of the high-resolution SBEM dataset to train DeepACSON to segment low-resolution large field-of-view SBEM datasets of white matter. We developed the CSD algorithm for the DeepACSON instance segmentation that decomposes tubular structures (myelinated axons) into their semantic
components (constituent axons). ACSON segmented thousands of myelinated axons, and DeepACSON segmented hundreds of thousands of myelinated axons with excellent evaluation scores.
The proposed ACSON pipeline was the first attempt to segment white matter ultrastructures in 3D EM datasets. The SRG-based pipeline was fully automated and required no human intervention for the segmentation. In the ACSON study, we reported an extensive and realistic 3D morphological analysis of white matter ultrastructures in SBEM volumes by analyzing the cross-sections perpendicular to the axonal skeleton. We measured non-trivial variations in the axonal diameter of myelinated axons, implying that the assumption of perfectly cylindrical axons can be simplistic for biophysical modeling purposes. In DeepACSON, we included the information about the shape of myelinated axons as a global objective for their 3D segmentation using the CSD algorithm as an alternative to the bottom-up
solutions for the instance segmentation of neuronal processes in the current state-of-the-art methodologies. We also trained DeepACSON to perform semantic segmentation, using datasets that were automatically segmented by the ACSON pipeline, proposing a method for human-annotation-free training. The CSD algorithm in the DeepACSON pipeline is a highly parallelizable technique, where this ability reduces the computation time of segmentation in large 3D SBEM datasets when tens of thousands of myelinated axons are tightly packed in a fraction of a cubic millimeter. In the DeepACSON study, we also emphasized a low-resolution yet-traceable SBEM imaging paradigm to resolve white matter
myelinated axons, as in return, increase the field of view. This EM image
acquisition strategy enabled quantifying parameters such as the axonal tortuosity or mitochondrial distance that had no precedent because they are more relevant
to large field-of-view imaging, indicating that low-resolution EM acquisition did not compromise the accuracy of measuring the axonal diameter. Finally, our
techniques offered new ways to understand the actual morphology of the brain and could capture ultrastructural alterations during disease; these kinds of quantification can make it possible to identify novel potential biomarkers which can be used to monitor progression in certain brain diseases.
REFERENCES
Abdollahzadeh, A. et al. (2019) ‘Automated 3D Axonal Morphometry of White Matter’, Scientific Reports. Nature Publishing Group, 9(1), p. 6084. doi:
10.1038/s41598-019-42648-2.
Abdollahzadeh, A. et al. (2021) ‘DeepACSON automated segmentation of white matter in 3D electron microscopy’, Communications Biology. Springer US, 4(1), p.
179. doi: 10.1038/s42003-021-01699-w.
Abdollahzadeh, A., Sierra, A. and Tohka, J. (2021) ‘Cylindrical Shape Decomposition for 3D Segmentation of Tubular Objects’, IEEE Access, 9, pp. 23979–23995. doi:
10.1109/ACCESS.2021.3056958.
Achanta, R. et al. (2010) ‘Slic Superpixels Technical Report’, EPFL Technical Report, 149300.
Adams, R. and Bischof, L. (1994) ‘Seeded region growing’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(6), pp. 641–647.
Alexander, D. C. et al. (2010) ‘Orientationally invariant indices of axon diameter and density from diffusion MRI’, NeuroImage. Elsevier Inc., 52(4), pp. 1374–1389. doi:
10.1016/j.neuroimage.2010.05.043.
Armstrong, R. C. et al. (2016) ‘White matter involvement after TBI: Clues to axon and myelin repair capacity’, Experimental Neurology. Elsevier B.V., 275, pp. 328–
333. doi: 10.1016/j.expneurol.2015.02.011.
Asafi, S., Goren, A. and Cohen-Or, D. (2013) ‘Weak convex decomposition by lines-of-sight’, Eurographics Symposium on Geometry Processing, 32(5), pp. 23–31. doi:
10.1111/cgf.12169.
Au, O. K. C. et al. (2008) ‘Skeleton extraction by mesh contraction’, SIGGRAPH’08:
International Conference on Computer Graphics and Interactive Techniques, ACM SIGGRAPH 2008 Papers 2008.
Baumeister, W. (2002) ‘Electron tomography: Towards visualizing the molecular organization of the cytoplasm’, Current Opinion in Structural Biology. doi:
10.1016/S0959-440X(02)00378-0.
Bear, M. F., Connors, B. W. and Paradiso, M. A. (2007) Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins.
Belevich, I. et al. (2016) ‘Microscopy Image Browser: A platform for segmentation and analysis of multidimensional datasets’, PLoS Biology, 14(1), pp. 1–13. doi:
10.1371/journal.pbio.1002340.
Berning, M., Boergens, K. M. and Helmstaedter, M. (2015) ‘SegEM: Efficient Image Analysis for High-Resolution Connectomics’, Neuron. Elsevier Inc., 87(6), pp.
1193–1206. doi: 10.1016/j.neuron.2015.09.003.
Berretti, S., Del Bimbo, A. and Pala, P. (2009) ‘3D Mesh decomposition using Reeb graphs’, Image and Vision Computing. Elsevier B.V., 27(10), pp. 1540–1554. doi:
10.1016/j.imavis.2009.02.004.
Bock, D. D. et al. (2011) ‘Network anatomy and in vivo physiology of visual cortical neurons’, Nature. Nature Publishing Group, 471(7337), pp. 177–184. doi:
10.1038/nature09802.
Caselles, V., Kimmel, R. and Sapiro, G. (1997) ‘Geodesic Active Contours’, International Journal of Computer Vision, 22(1), pp. 61–79 (1997). doi:
https://doi.org/10.1023/A:1007979827043.
Chan, T. F. and Vese, L. A. (2001) ‘Active contours without edges’, IEEE Transactions on Image Processing, 10(2), pp. 266–277. doi: 10.1109/83.902291.
Chazal, F., Lieutier, A. and Rossignac, J. (2005) ‘Projection-homeomorphic surfaces’, ACM Symposium on Solid Modeling and Applications, SM, pp. 9–14. doi:
10.1145/1060244.1060247.
Chen, X., Golovinskiy, A. and Funkhouser, T. (2009) ‘A Benchmark for {3D} Mesh Segmentation’, ACM Transactions on Graphics (Proc. SIGGRAPH), 28(3).
Chen, X. H. et al. (2009) ‘A lack of amyloid β plaques despite persistent
accumulation of amyloid β in axons of long-term survivors of traumatic brain injury’, Brain Pathology, 19(2), pp. 214–223. doi:
10.1111/j.1750-3639.2008.00176.x.
Chomiak, T. and Hu, B. (2009) ‘What is the optimal value of the g-ratio for
myelinated fibers in the rat CNS? A theoretical approach’, PLoS ONE, 4(11). doi:
10.1371/journal.pone.0007754.
Cohen, L. D. (1991) ‘On active contour models and balloons’, CVGIP: Image Understanding. doi: 10.1016/1049-9660(91)90028-N.
Cornea, N. D., Silver, D. and Min, P. (2007) ‘Curve-skeleton properties, applications, and algorithms’, IEEE Transactions on Visualization and Computer Graphics, 13(3), pp. 530–548. doi: 10.1109/TVCG.2007.1002.
Czum, J. M. (2020) ‘Dive Into Deep Learning’, Journal of the American College of Radiology. doi: 10.1016/j.jacr.2020.02.005.
Deerinck, T. et al. (2010) ‘Enhancing serial block-face scanning electron microscopy to enable high resolution 3-D nanohistology of cells and tissues’, Microscopy and Microanalysis. KYS tieteellinen kirjasto, 16(Suppl 2), pp. 1138–1139. doi:
10.1017/S14319276100.
Denk, W. and Horstmann, H. (2004) ‘Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure’, PLoS Biology. Edited by Kristen M. Harris, 2(11), p. e329. doi:
10.1371/journal.pbio.0020329.
Douglas Fields, R. (2008) ‘White Matter Matters’, Scientific American, 298(3), pp. 54–
61. doi: 10.1038/scientificamerican0308-54.
Duchatel, R. J., Shannon Weickert, C. and Tooney, P. A. (2019) ‘White matter neuron biology and neuropathology in schizophrenia’, npj Schizophrenia. Springer US, 5(1). doi: 10.1038/s41537-019-0078-8.
Duchi, J., Hazan, E. and Singer, Y. (2011) ‘Adaptive subgradient methods for online learning and stochastic optimization’, Journal of Machine Learning Research.
Falk, T. et al. (2019) ‘U-Net: deep learning for cell counting, detection, and
morphometry’, Nature Methods, 16(1), pp. 67–70. doi: 10.1038/s41592-018-0261-2.
Fan, J. et al. (2005) ‘Seeded region growing: an extensive and comparative study’, Pattern Recognition Letters, 26(8), pp. 1139–1156. doi:
10.1016/j.patrec.2004.10.010.
Farabet, C. (2013) ‘Towards Real-Time Image Understanding with Convolutional Networks’, PhD Dissertation.
Funke, J. et al. (2019) ‘Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction’, IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE, 41(7), pp. 1669–1680. doi:
10.1109/TPAMI.2018.2835450.
Ginsburger, K. et al. (2019) ‘MEDUSA: A GPU-based tool to create realistic phantoms of the brain microstructure using tiny spheres’, NeuroImage, 193(March), pp. 10–24. doi: 10.1016/j.neuroimage.2019.02.055.
Goyal, M. et al. (2012) ‘Towards locally and globally shape-aware reverse 3D modeling’, CAD Computer Aided Design. Elsevier Ltd, 44(6), pp. 537–553. doi:
10.1016/j.cad.2011.12.004.
Gray, E. G. (1959) ‘Axo-somatic and axo-dendritic synapses of the cerebral cortex’, Journal of Anatomy, 93(Pt 4), pp. 420–433.
Haberl, M. G. et al. (2018) ‘CDeep3M—Plug-and-Play cloud-based deep learning for image segmentation’, Nature Methods. Springer US, 15(9), pp. 677–680. doi:
10.1038/s41592-018-0106-z.
Hassouna, M. S. and Farag, A. A. (2005) ‘Robust Centerline Extraction Framework Using Level Sets’, in 2005 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (CVPR’05). IEEE, pp. 458–465. doi:
10.1109/CVPR.2005.306.
Hayworth, K. et al. (2006) ‘Automating the collection of ultrathin serial sections for large volume TEM reconstructions’, Microscopy and Microanalysis, 12(S02), pp.
86–87. doi: 10.1017/S1431927606066268.
He, K. and Sun, J. (2015) ‘Convolutional neural networks at constrained time cost’, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 5353–5360. doi: 10.1109/CVPR.2015.7299173.
Hoppe, W. (1981) ‘Three-Dimensional Electron Microscopy’, Annual Review of Biophysics and Bioengineering, 10(1), pp. 563–592. doi:
10.1146/annurev.bb.10.060181.003023.
Januszewski, M. et al. (2018) ‘High-precision automated reconstruction of neurons with flood-filling networks’, Nature Methods. Springer US, 15(8), pp. 605–610.
doi: 10.1038/s41592-018-0049-4.
Johnson, V. E., Stewart, W. and Smith, D. H. (2013) ‘Axonal pathology in traumatic brain injury’, Experimental Neurology. Elsevier Inc., 246, pp. 35–43. doi:
10.1016/j.expneurol.2012.01.013.
Jorstad, A. and Fua, P. (2015) ‘Refining Mitochondria Segmentation in Electron Microscopy Imagery with Active Surfaces’, in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 367–379. doi: 10.1007/978-3-319-16220-1_26.
Kaick, O. Van et al. (2015) ‘Shape Segmentation by Approximate Convexity Analysis’, ACM Transactions on Graphics, 34(1), pp. 1–11. doi: 10.1145/2611811.
Kamiya, K. et al. (2017) ‘Diffusion imaging of reversible and irreversible microstructural changes within the corticospinal tract in idiopathic normal pressure hydrocephalus’, NeuroImage: Clinical. The Authors, 14, pp. 663–671.
doi: 10.1016/j.nicl.2017.03.003.
Kass, M., Witkin, A. and Terzopoulos, D. (1988) ‘Snakes: Active contour models’, International Journal of Computer Vision, 1(4), pp. 321–331. doi:
Kass, M., Witkin, A. and Terzopoulos, D. (1988) ‘Snakes: Active contour models’, International Journal of Computer Vision, 1(4), pp. 321–331. doi: