While annotating the dataset, it was noticed that except the feed roller periodic damages there are also non-periodic mechanical damages. In order to reveal more details of the CNN architectures defect detection abilities, the following experiments were carried out:
1. Classification of all mechanical damages treated as a single class.
2. Classification of feed roller periodic traces only.
The purpose of first experiment is to estimate the ability of the selected CNN architec-ture to distinguish the mechanically damaged parts of the board regardless of the damage type. Non-periodic damages may significantly affect the accuracy because they are more variable in appearance and less common in the existing dataset. That is why the second experiment was carried out to estimate the CNN performance to classify periodic dam-ages since they are more regular. Finally, the last experiment measures the ability of the network not only to detect the damage, but also to distinguish its type.
All datasets including annotations were generated with MATLAB [47]. The CNN archi-tectures were trained with the Caffe [48] deep learning framework. The performance was evaluated with the scripts for Python 3.6 programming language with Caffe module. To reduce training time and increase classification accuracy, the transfer learning approach was applied. The CNNs were initialized with the pretrained models from another Master’s thesis [49] within the DigiSaw project, because of the similarity of the used images.
5.4 Results
5.4.1 Experiment 1: classification of the combined mechanical damages
The performance of localization and binary classification of combined periodic and non-periodic mechanical damages to a single class is provided in Table 1. The confusion matrices for each of the trained CNN architectures are shown in Figure 36.
Table 1. Performance comparison of different CNN architectures in case of combined binary classification.
5.4.2 Experiment 2: classification of the feed roller periodic traces
The performance of localization and binary classification of feed roller periodic mechan-ical damages only is provided in Table 2. The confusion matrices for each of the trained CNN architectures are shown in Figure 37. The percentage of images containing feed roller mechanical defects and the defects were detected correctly with respect to the Jac-card metric threshold for every CNN architecture is shown in Figure 38. Examples of feed rollers periodic damages detection results with GoogLeNet CNN architecture are given in Figure 39. Comparison of the results obtained for the same image with every trained CNN is shown in Figures 40 and41.
(a) (b)
(c) (d)
Figure 36. Confusion matrices of CNN architectures trained for combined binary classification:
(a) AlexNet, (b) GoogLeNet, (c) VGG-16, (d) ResNet-50.
5.4.3 Inference time
The most time consuming stage of the proposed method is the sequential patches classifi-cation with the CNN. The lower inference time consumed by a single patch classificlassifi-cation, the more effective the given CNN architecture. Inference time for all CNNs was mea-sured on MSI GE70 laptop with Intel Core i7-4700MQ @ 2.4 GHz processor, NVIDIA GeForce GTX 760M single GPU, and Ubuntu 17.10 operating system.
Table 3 contains average time required to classify one patch of the board image for se-lected CNNs.
Table 2. Performance comparison of different CNN architectures in case of feed rollers periodic mechanical damages binary classification.
Architecture Accuracy SJaccard
AlexNet 0.857 0.384
GooLeNet 0.907 0.580
VGG-16 0.927 0.588
ResNet-50 0.913 0.546
Table 3.Single patch average inference time for each of the trained CNN architectures.
Architecture Inference time, seconds
AlexNet 0.022
GoogLeNet 0.023
VGG-16 0.101
ResNet-50 0.051
(a) (b)
(c) (d)
Figure 37. Confusion matrices of CNN architectures trained for feed rollers periodic mechanical damages binary classification: (a) AlexNet, (b) GoogLeNet, (c) VGG-16, (d) ResNet-50.
Figure 38. Percentage of images with correctly detected feed roller mechanical defects with respect to the Jaccard metric threshold.
Figure 39. Several examples of feed roller periodic mechanical damages detection with GoogLeNet CNN architecture. Ground truth damages are marked with red bounding boxes, while the predicted defective parts of boards are highlighted with green color.
(a)
(b)
(c)
(d)
Figure 40.Example of detection of the same slight feed roller periodic mechanical damages with:
(a) AlexNet; (b) GoogLeNet; (c) VGG-16; (d) ResNet-50. Ground truth damages are marked with red bounding boxes, while the predicted defective parts of boards are highlighted with green color.
(a)
(b)
(c)
(d)
Figure 41.Examples of detection of the same hard feed roller periodic mechanical damages with:
(a) AlexNet; (b) GoogLeNet; (c) VGG-16; (d) ResNet-50. Ground truth damages are marked with red bounding boxes, while the predicted defective parts of boards are highlighted with green color.
6 DISCUSSION
6.1 Current study
Mechanical damages on sawn timber can appear during the sawmilling process. Mechani-cal damages can be caused by the feed rollers of the sawing machine or by other excessive interactions with the board surfaces. Those damages significantly affect the quality and the price of the particular board, therefore it is crucial to detect them. Based on the lit-erature review, it can be said that the automatic detection of mechanical damages has not been studied. The majority of the existing solutions in sawn timber surface inspection solves the problem of classification and detection natural damages such as knots, worm holes, watermarks and fungus. However, the mechanical damages are no less important.
For the purpose of the mechanical damages detection, the dataset of images of 127 sawn timber boards was annotated with bounding boxes and labels specific for different defect types including natural. The proposed method segments the board from the background, splits the segmented board into the overlapping patches, classify them and, finally, localize the mechanical defects according to the classification results. This database of the board patches was used to train four CNN architectures: AlexNet, GoogLeNet, VGG-16 and ResNet-50 to distinguish the defective and normal regions of the boards. Because of the limited number of the board images, and the significant variability of the defects width to height ratio, the 1-dimension sliding window detection technique was used instead of straightforward detection on a single image with a state-of-the-art CNN-architectures such as Faster R-CNN, YOLO or R-FCN.
The proposed solution achieved very promising individual patch classification accuracy of more than 92% for VGG-16 architecture. At the same time the GoogLeNet archi-tecture achieved the lowest false negative and false positive rates both of 9% and the GoogLeNet architecture is four time faster in single patch processing than the VGG-16.
Both GoogLeNet and VGG-16 architectures have shown the best detection accuracy with the Jaccard coefficient greater than 0.58. The worst classification and detection accuracy (85.7% and 0.384) was shown by AlexNet architecture. Also, AlexNet showed the worst false positive rate of 41%. It could be explained by the low number of convolutional lay-ers in AlexNet. The possible reason for the similar results of the VGG-16 and GoogLeNet architectures is the fact that the VGG-16 uses only 3×3kernels, while the GoogLeNet uses the combination of1×1,3×3and5×5kernels in every inception layer. It means that the GoogLeNet can degrade to the architecture similar to the VGG-16 during training
stage. The main drawback of the proposed method is a large computational time needed to process all the images of a single board, since those images are split into patches and are fed sequentially to the CNN input.
6.2 Future work
The main possible objectives for the future work include accuracy improvement and re-quired time reduction. Accuracy can be improved by using a larger image dataset labeled by the experts of the sawmill industry. To reduce the computational costs, the single image end-to-end CNN-based architecture could be used. However, the large variability of the defect sizes and large width to height ratio of board images should be taken into account.
Finally, the combined automated surface inspection system that would be able to detect and distinguish any kind of defects both mechanical and natural can be implemented.
7 CONCLUSION
This study introduced the problem of the mechanical damage detection on the surface of sawn timber boards. The existing methods applied for timber inspection were reviewed.
The principals of convolutional neural networks and their adaptations for object localiza-tion were surveyed since the CNN-based approaches are the most promising in the image classification tasks.
In this thesis the method for mechanical damages detection on sawn timber images was introduced. The proposed method segments the board on the image, splits the part of the image containing the board into overlapping patches, classifies the patches with the CNN, and, finally, determines the defect location based on classification results and the coordinates of patches. The experimental part of the work contained the performance comparison of four CNN architectures: AlexNet, GoogLeNet, VGG-16, and ResNet-50.
The VGG-16 architecture produced the best results with a very promising classification accuracy of more than 92% for individual patches.
REFERENCES
[1] Yrjö Nuutinen, Kari Väätäinen, Antti Asikainen, Robert Prinz, and Jaakko Heinonen. Operational efficiency and damage to sawlogs by feed rollers of the harvester head. Silva Fennica, 44(1):121–139, 2010.
[2] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C.
Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Interna-tional Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
[3] Digisaw — leap of digitalisation for sawmill industry. http://www2.it.lut.
fi/project/digisaw/index.shtml. [Online; accessed May 25, 2018].
[4] UR Hashim, SZ Hashim, and AK Muda. Automated vision inspection of timber surface defect: a review. Jurnal Teknologi, 77(20):127–135, 2015.
[5] Hau Lee Tong, Hu Ng, Tzen Vun Timothy Yap, Wan Siti Halimatul Munirah Wan Ahmad, and Mohammad Faizal Ahmad Fauzi. Evaluation of feature extraction and selection techniques for the classification of wood defect images. Journal of Engi-neering and Applied Sciences, 12(3):602–608, 2017.
[6] SK Sharma and SR Shukla. Properties evaluation and defects detection in timbers by ultrasonic non-destructive technique. Journal of the Indian Academy of Wood Science, 9(1):66–71, 2012.
[7] Shan Gao, Na Wang, Lihai Wang, and Jingquan Han. Application of an ultrasonic wave propagation field in the quantitative identification of cavity defect of log disc.
Computers and Electronics in Agriculture, 108:123–129, 2014.
[8] Fleur Longuetaud, Frédéric Mothe, Bertrand Kerautret, Adrien Krähenbühl, Laurent Hory, Jean Michel Leban, and Isabelle Debled-Rennesson. Automatic knot detec-tion and measurements from X-ray CT images of wood: a review and validadetec-tion of an improved algorithm on softwood samples. Computers and Electronics in Agri-culture, 85:77–89, 2012.
[9] Erik Johansson, Dennis Johansson, Johan Skog, and Magnus Fredriksson. Auto-mated knot detection for high speed computed tomography on pinus sylvestris l.
and picea abies (l.) karst. using ellipse fitting in concentric surfaces.Computers and Electronics in Agriculture, 96:238–245, 2013.
[10] A Wyckhuyse and Xavier Maldague. A study of wood inspection by infrared ther-mography, part i: Wood pole inspection by infrared thermography. Journal of Re-search in Nondestructive Evaluation, 13(1):1–12, 2001.
[11] Gamaliel López, L-Alfonso Basterra, Gemma Ramón-Cueto, and Agustín de Diego.
Detection of singularities and subsurface defects in wood by infrared thermography.
International Journal of Architectural Heritage, 8(4):517–536, 2014.
[12] Olle Hagman. Multivariate prediction of wood surface features using an imaging spectrograph. Holz als Roh-und Werkstoff, 55(6):377–382, 1997.
[13] Ruoxu Ren, Terence Hung, and Kay Chen Tan. A generic deep-learning-based approach for automated surface inspection. IEEE Transactions on Cybernetics, 48(3):929–940, 2018.
[14] Timo Ojala, Matti Pietikäinen, and David Harwood. A comparative study of texture measures with classification based on featured distributions. Pattern Recognition, 29(1):51–59, 1996.
[15] Robert M Haralick, Karthikeyan Shanmugam, and Its’hak Dinstein. Textural fea-tures for image classification.IEEE Transactions on Systems, Man, and Cybernetics, SMC-3(6):610–621, 1973.
[16] David G Lowe. Object recognition from local scale-invariant features. In Proceed-ings of the 7th International Conference on Computer Vision (ICCV), volume 2, pages 1150–1157. IEEE, 1999.
[17] Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Speeded-up robust features SURF.Computer Vision and Image Understanding, 110(3):346–359, 2008.
[18] Hideyuki Tamura, Shunji Mori, and Takashi Yamawaki. Textural features corre-sponding to visual perception. IEEE Transactions on Systems, Man, and Cybernet-ics, 8(6):460–473, 1978.
[19] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wo-jna. Rethinking the inception architecture for computer vision. InProceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–
2826. IEEE, 2016.
[20] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: a deep convolutional activation feature for generic visual recognition. InProceedings of the 31st International Conference on Machine Learning (ICML), volume 32, pages 647–655. PMLR, 2014.
[21] Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, 1979.
[22] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image seg-mentation. International Journal of Computer Vision, 59(2):167–181, 2004.
[23] Young-Jin Cha, Wooram Choi, and Oral Büyüköztürk. Deep learning-based crack damage detection using convolutional neural networks. Computer-Aided Civil and Infrastructure Engineering, 32(5):361–378, 2017.
[24] Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient-based learning. InShape, Contour and Grouping in Computer Vision, pages 319–345. Springer, 1999.
[25] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541–551, 1989.
[26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. InProceedings of the 25th Conference on Neural Information Processing Systems (NIPS), pages 1097–1105, 2012.
[27] Saleh Albelwi and Ausif Mahmood. A framework for designing the architectures of deep convolutional neural networks. Entropy, 19(6):242, 2017.
[28] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning.IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2010.
[29] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
[30] Moojoon Shin and Jee-Hyong Lee. Accurate lithography hotspot detection using deep convolutional neural networks. Journal of Micro/Nanolithography, MEMS, and MOEMS, 15(4):043507, 2016.
[31] Yue Liu, Jun Yong, Liang Liu, Jinlong Zhao, and Zongyu Li. The method of insula-tor recognition based on deep learning. In4th International Conference on Applied Robotics for the Power Industry (CARPI), pages 1–5. IEEE, 2016.
[32] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9. IEEE, 2015.
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. InProceedings of the International Conference on Learning Representations (ICLR), 2014.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, 2016.
[35] Bin Wang, Sheng Tang, Ruizhen Zhao, Wu Liu, and Yigang Cen. Pedestrian detec-tion based on region proposal fusion. In17th International Workshop on Multimedia Signal Processing (MMSP), pages 1–6. IEEE, 2015.
[36] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hier-archies for accurate object detection and semantic segmentation. InProceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 580–
587. IEEE, 2014.
[37] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeul-ders. Selective search for object recognition. International Journal of Computer Vision (IJCV), 104(2):154–171, 2013.
[38] Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV), pages 1440–1448. IEEE, 2015.
[39] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. InProceedings of the 28th Conference on Neural Information Processing Systems (NIPS), pages 91–99, 2015.
[40] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN.
InProceedings of the International Conference on Computer Vision (ICCV), pages 2980–2988. IEEE, 2017.
[41] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. InProceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440. IEEE, 2015.
[42] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788. IEEE, 2016.
[43] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: Object detection via region-based fully convolutional networks. InProceedings of the 29th Conference on Neu-ral Information Processing Systems (NIPS), pages 379–387, 2016.
[44] Shane Torbert. Applied Computer Science. Springer, 2016.
[45] Robert M Haralick and Linda G Shapiro. Computer and Robot Vision. Addison-Wesley, 1992.
[46] Kasthurirangan Gopalakrishnan, Siddhartha K Khaitan, Alok Choudhary, and Ankit Agrawal. Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Construction and Building Materials, 157:322–330, 2017.
[47] MATLAB. version 9.4.0 (R2018a). The MathWorks Inc., Natick, Massachusetts, 2018.
[48] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. InProceedings of the 22nd International Conference on Multimedia, pages 675–678. ACM, 2014.
[49] Dmitrii Shustrov. Species identification of wooden material using convolutional neural networks. Master’s thesis, Lappeenranta University of Technology, Finland, 2018.