RGB Camera-Based Imaging of Oxygen Saturation and Hemoglobin Concentration in Ocular Fundus

(1)

Rinnakkaistallenteet Luonnontieteiden ja metsätieteiden tiedekunta

2019

RGB Camera-Based Imaging of

Oxygen Saturation and Hemoglobin Concentration in Ocular Fundus

Nakano, Kazuya

Institute of Electrical and Electronics Engineers (IEEE)

Tieteelliset aikakauslehtiartikkelit

CC BY http://creativecommons.org/licenses/by/4.0/

http://dx.doi.org/10.1109/ACCESS.2019.2913878

https://erepo.uef.fi/handle/123456789/7661

Downloaded from University of Eastern Finland's eRepository

(2)

RGB Camera-Based Imaging of Oxygen Saturation and Hemoglobin Concentration in Ocular Fundus

KAZUYA NAKANO ¹, RYOSUKE HIROFUJI², TAKASHI OHNISHI¹, MARKKU HAUTA-KASARI³, IZUMI NISHIDATE⁴, AND HIDEAKI HANEISHI¹

1Center for Frontier Medical Engineering, Chiba University, Chiba 263-8522, Japan 2NIDEK, Co., Ltd., Gamagori, Aichi 443-0038, Japan

3School of Computing, University of Eastern Finland, Joensuu, Finland

4Graduate School of Bio-Applications and Systems Engineering, Tokyo University of Agriculture and Technology, Koganei, Tokyo 184-8588, Japan

Corresponding author: Kazuya Nakano (knakano@chiba-u.jp) This work was supported by the JSPS Core-to-Core Program.

ABSTRACT We propose a red, green, blue (RGB)-based oximetry to assess the ocular fundus and determine its oxygen saturation (SO₂) and hemoglobin concentration. The oxygenated hemoglobin concentration, deoxygenated hemoglobin concentration, and SO₂were estimated employing a method that combines Monte Carlo simulation of light transport in the fundus tissue with a multiple regression analysis. In this study, a single-layer model of the ocular fundus was employed for the Monte Carlo simulation. We constructed an experimental apparatus for measuring the fundus of a rat’s eye using an RGB detector and investigated the physiological response that occurs upon a change in the fraction of inspired oxygen (FiO2). The resultant images of oxygenated hemoglobin concentration, deoxygenated hemoglobin concentration, total hemoglobin concentration, and SO2indicated that the response was caused by the defective oxygenation of the blood. The results of the present study indicate the possibility of oximetry based on the RGB images of a fundus.

INDEX TERMS Digital cameras, biomedical optical imaging, image color analysis.

I. INTRODUCTION

Observations of the ocular fundus are useful for diagnosing fundus diseases such as age-related macular degeneration (AMD) [1], diabetic retinopathy [2], [3], retinal pigmen- tary degeneration [4], and retinal vascular occlusions [5].

Fig. 1 shows the ocular fundus of a rat that was used in this study. We can observe the vessels in the ocular fundus directly because some parts of the eyeball, such as the vit- reous body and the crystalline lens, are transparent. There- fore, observations of vessels in the fundus facilitate the diag- noses of fundus disease and various other diseases related to blood vessels in the eye, such as hypertension [6] and kidney disease [7]. Currently, doctors diagnose these diseases qualitatively by observing pictures taken with a fundus camera. The fundus has various pigments that absorb light of different wavelengths [8]. Some studies have reported that target pigments or tissues of the retina can be emphasized by selecting an optimal wavelength from multichannel spectral images of the retina [9], [10]. However, it is difficult to extract the specific information of the retina directly from the red,

The associate editor coordinating the review of this manuscript and approving it for publication was Sudhakar Radhakrishnan.

green, blue (RGB) image acquired by the fundus camera because each channel of the RGB image integrates over a broad spectral band based on the transmittance spectrum of each color filter. Moreover, fundus fluorescein angiography (FFA) [11], [12] or indocyanine green angiography (ICGA) [13], [14] has also been widely used to diagnose eye diseases, such as diabetic retinopathy and retinal vascular occlusions.

However, the imaging agent may cause side effects, such as nausea, itchy red areas, cough,etc. Rare cases of anaphylactic shock have also been reported. Because of the heavy bur- den FFA places on patients, it is desirable to quantitatively evaluate the vessels of the ocular fundus for early detection of these diseases without administering an imaging agent to subjects. One of the signs of these diseases is the forma- tion of new blood vessels in the ocular fundus caused by hypoxia [15]. Moreover, the retinal vessel occlusion causes decreased oxygen saturation (SO₂) [16]. Therefore, SO₂ in the ocular fundus allows for early detection of these diseases.

Many researchers have proposed various retinal SO₂imaging methods. An initial study in this area was conducted by Hickamet al.[17], where fundus pictures were acquired through two filters, which passed light of wavelengths 640 nm and 800 nm. Then, the venous SO₂on the optic disc

VOLUME 7, 2019

2169-35362019 IEEE. Translations and content mining are permitted for academic research only.

Personal use is also permitted, but republication/redistribution requires IEEE permission. 56469

(3)

FIGURE 1. Structure of rat eye. (a) Sagittal plane. (b) Coronal plane.

was estimated from a linear relationship between SO₂and the ratio of the red to infrared optical density (OD) images.

Additionally, other studies have reported two-wavelength oximetry [18]–[23]. Two commercial two-wavelength retinal oximetry systems, Vesselmap oxygen module (Imedos Sys- tems GmbH, Jena, Germany) [21] and Oxymap T1 (Oxyma ehf., Reykjavik, Iceland) [22], have contributed to ophthal- mological research. Recently, hyperspectral imaging (HSI) [24]–[26], [34], [35] and multispectral imaging (MSI) [25], [27]–[33], [36], [37], [34], [40]–[46] have also been used to estimate SO2 in the fundus. HSI can be used to acquire spectral images with higher wavelength resolution than that of MSI but is limited to time-sequential measurements.

Alternatively, MSI can be used to acquire images from a snapshot that can concurrently acquire images at multiple wavelengths or by time-sequential imaging. Either of the two band-pass interference filters, one a liquid-crystal tunable filter (LCTF) and the other an acousto-optical tunable filter (AOTF), was used in the time-sequential measurements.

Seven-band MSI-based oximetry with LCTF, which has a wavelength sensitivity range from 522 to 586 nm, was proposed by Khoobeiet al. [33]. HSI-based oximetry with LCTF for the human retina was developed later [24], [34], [35].

Five-band MSI-based oximetry techniques with AOTFs were also proposed [25], [36], and HSI-based oximetry with AOTFs have also been developed [25], [26]. AOTFs can switch wavelengths faster and have narrower bandwidths than LCTFs. Further, multispectral scanning laser ophthal- moscopes (SLO), which have higher contrast and spatial resolution than fundus cameras, were developed for oximetry [27]–[32], [37]. These techniques illuminate and scan the retina using multiple lasers of different wavelengths. However, these time-sequential oximetry methods are affected by involuntary eye movement, which can be classified as microsaccade, drift, and tremor [38]. Indeed, Hendargoet al. reported that the snapshot MSI could reduce the motion artifact that was caused by the blood flow in the blood vessels [39]. Moreover, MacKenzieet al. observed the rapid oxygen dynamics using the snapshot MSI [40]. There-

fore, to eliminate the influence of involuntary eye movement during measurement, the snapshot MSI technique is more suitable for the estimation of SO₂ in the fundus, because this technique allows multiband images to be acquired simul- taneously without switching wavelengths. Currently, there are two types of snapshot MSI that have been used for retinal oximetry. The first type uses beam-splitter multiplexing, which splits a single beam into two or more, then spectrally filters the split beams [41], [42]. While this approach is simple, it is optically inefficient because a beam is being split by multiple beam splitters. The second approach is the use of an image replicating imaging spectrometer (IRIS), which demultiplexes light with quarter wave plates and Wollaston prisms. Alabboud et al. first reported basic data on snap- shot MSI oximetry using IRIS [34]. After that, applications including retinal oximetry that utilized IRIS were reported by other researchers [43]–[46]. Moreover, some researchers have reported that photoacoustic imaging (PAI) [47] and optical coherence tomography (OCT) [48] could also been used to estimate SO₂ in the retina. However, these systems require complex optical systems.

To address these problems, we propose a noninvasive imaging method for measuring SO₂ based on diffuse reflectance spectroscopy with an RGB color detector that is mounted on commercial fundus cameras. Therefore, our method requires no additional equipment and can be used to estimate SO2 in the fundus from only an RGB image.

Since the RGB color detector is less affected by the involuntary eye movement, it is suitable for fundus measurement. This technique is based on a method that combines a Monte Carlo simulation (MCS) of light transport in the fundus tissue with multiple regression analysis (MRA). In this study, a single-layer model of the ocular fundus was used for the MCS. Moreover, to evaluate the hemodynamics in the ocular fundus, we formed SO₂ images from the ocular fundus images of a rat by changing the fraction of inspired oxygen (FiO₂).

There are two previous studies of oximetry that have employed an RGB charge-coupled device (CCD) or com- plementary metal-oxide-semiconductor (CMOS) image sensor [21], [49]. Hammeret al. designed two monochromatic fundus cameras using specific dual wavelength transition filters (548, and 610 nm) and a color CCD detector [21].

Moreover, Puttenet al. designed a multispectral microscope that combined LCTF and a digital single-lens reflex (SLR) camera [49]. However, our method is different from these studies. That is, our method requires only an RGB detector and no additional equipment, whereas these other systems require special filters. In the RGB image sensor, red, green, blue filters are arranged on the pixel array based on a Bayer arrangement [50]. The Bayer filter consists of 50% green filters, 25% red filters, and 25% blue filters [50].

Retinal oximetry based on the RGB image or MSI can detect the initial state of disease from oxygen dynamics.

Alternatively, ICGA can detect the state of deep vessels that are difficult to observe using visible light. This is

(4)

FIGURE 2. Single-layer model of the ocular fundus and the molar extinction coefficient spectra of oxygenated hemoglobin and deoxygenated hemoglobin (from Ref. [61]).

because ICGA generates fluorescent light at a wavelength of 800-850 nm and this near-infrared (NIR) light has high transmissivity in a living body [51]. Therefore, retinal oximetry and ICGA can complement each other.

II. PRINCIPLES

A. RELATIONSHIP BETWEEN RGB VALUES AND HEMOGLOBIN CONCENTRATION IN

THE OCULAR FUNDUS

The RGB values of a fundus image captured using a fundus camera can be expressed as follows:



 R G B



=L₁



 X Y Z



, (1) where X, Y, and Z are Commission Internationale de l’Éclairage XYZ (CIE XYZ) tristimulus values. Based on the result of the RGB color model of human eye, the XYZ, which do not correspond to any physical spectrum, were mathematically determined by CIE in 1931 [52]. L₁ exists for each RGB working space of the National Television Stan- dards Committee (NTSC), the standard RGB (sRGB),etc., and is a transformation matrix to convert XYZ values to the corresponding RGB values. These XYZ values are defined as

X =kX

λ

E(λ)x¯(λ)O(λ) ,

Y =kX

λ

E(λ)y¯(λ)O(λ) ,

Z =kX

λ

E(λ)z¯(λ)O(λ) , (2) where E(λ), O(λ), and λ are the spectral distribution of the illuminant, diffuse reflectance spectrum of the fundus, and wavelength, respectively. The functionsx¯(λ),y¯(λ), and z¯(λ) are color-matching functions in the CIEXYZ color system [52], [53], andk is a constant whose value results in Y = 100 for a perfect diffuser.

k=100 X

λ

E(λ)y¯(λ) . (3) Fig. 2 shows the single-layer model of the ocular fundus constructed for this study. To simplify the model, it was assumed that the model has oxygenated and deoxygenated

hemoglobin only. The fundus has melanin and a yellow coloring matter (xanthophyll) [54]–[56], but the model in this study has none of these pigments. As Fig. 2 shows, incident light is scattered by the tissue and absorbed by the hemoglobin after incident light migrates into the tissue. Then, light exits from the tissue as diffuse reflected light. The diffuse reflectance of the retina is given by [57]

O(λ)= I I₀ =

Z ∞ 0

P l_r;µs,r,g_r

×exp

− µa,HbO(λ)+µa,HBR(λ) lr dlr

= Z ∞

0

P l_r;µs,r,g_r

×exp{−(C_HbOεHBO(λ)+C_HBRεHBR(λ))l_r}dl_r, (4) where I₀ and I represent the incident and reflected light intensities, respectively.P(l;µs,g)is the probability function of photon path length l, and it is determined by the scattering coefficientµsand anisotropy factorg. In addition, µa, C, and ε are the absorption coefficient, concentration, and extinction coefficient of the oxygenated and deoxygenated hemoglobin, respectively [58]. The subscripts, r, HbO, andHbRdenote retina, oxygenated hemoglobin, and deoxygenated hemoglobin, respectively. Herein, the absorption coefficientµa can be expressed as the product of the concentration and extinction coefficient, i.e., µa = Cε. Therefore, from Eq. (1) to (4), the RGB values are functions of the concentrationC.

B. ESTIMATION OF OCULAR FUNDUS CHROMOPHORE FROM RGB VALUES

If the path length probability function P(l;µs,g) can be measured, the concentration C can be calculated from the RGB values in Eq. (1) through (4). However, it is difficult to measure the function directly. Therefore, in this study, the MCS of the light transport is used to estimate the diffuse reflectance spectrum of the fundusO(λ).

Fig. 3 illustrates the procedure that was proposed by Nishidate et al. [60] for estimating C_HbO, C_HbR, C_HbT, and SO2 from the red-green-blue image. In this method, we need to calculate two transformation matrices, defined as N1 and N2, in advance. The first matrix N1 transforms RGB values into CIEXYZ tristimulus values, and the second matrix N2transforms CIEXYZ values intoCHbO andCHbR. After the white balance of the RGB detector is calibrated using a standard white diffuser with 99% reflectance as a reference material, the images of both a standard color chart consisting of 24 color chips and a standard white diffuser are acquired using the RGB detector. Moreover, to correct illuminance non-uniformity, the RGB values of each color chip are normalized by the RGB values of the white images.

The transformation matrix N₁relates RGB and XYZ as



 X Y Z



=N1



 R G B



, (5)

(5)

FIGURE 3. Estimation method of oxygenated hemoglobin concentrationC_HbO, deoxygenated hemoglobin concentration C_HbR, total hemoglobin concentrationC_HbT, and oxygen saturation SO₂from a red-green-blue image.

and is determined using the MRA between the measured RGB and the XYZ values. The XYZ values under specific illumination, which are supplied by the chart, are the response variables, while the measured RGB values are the predictor variables. The resulting regression coefficients are the elements of N₁. Even if the spectral characteristic of a color filter on the RGB detector and an illumination are unknown, any RGB values are transformed into the common XYZ values using the matrix N₁. Therefore, because of the matrix N₁, we do not need to consider the differences of individual devices. If we use another fundus camera system for estimation, our method requires measurement of the chart and calculation of N1on one occasion only.

Next, the second transformation matrix N2 is also determined using the MRA between the XYZ values and the chromophore concentrations by

C_HbO C_HbR

=N₂





 X Y Z 1





. (6) The chromophore concentrations, i.e., the oxygenated hemoglobin concentration C_HbO and deoxygenated hemoglobin concentrationC_HbR, represent the light absorption coefficients of the fundus. The C_HbO and C_HbR are the response variables, while the XYZ values are the predictor variables. The resulting regression coefficients are

the elements of N2. Using various combinations of these hemoglobin concentrations, we employ the MCS model for light transport in the fundus to calculate the diffuse reflectance spectra of the fundusO(λ). The parameters of the MCS are as follows. As mentioned before, the absorption coefficient µa(λ), which indicates the light absorption of the oxygenated and deoxygenated hemoglobin, is obtained from the concentrations C_HbO and C_HbR and the known extinction coefficient spectra ofεHbO andεHbR [59]. Then, the reduced scattering coefficientµ⁰_s(λ)can be approximated by [61], [62]

µ⁰_s(λ)=aλ^−b, (7) where the two parametersaandbare defined as the scattering amplitude and scattering power, respectively. The measured scattering spectrum of biological tissue gives the typical parameters. The coefficientsµ⁰_s(λ) are set by changing the typical parameters, a and b, in five steps [63], [64]. Five different parameters are derived by multiplying the typical parameters ofaandbby 0.5, 0.75, 1.0, 1.25, and 1.5, respectively. In addition, the refractive index [65] and layer thick- ness are set to be 1.4 and 5 cm, respectively. To determine matrix N₂, we use the simulation to calculate 300 diffuse reflectance spectra in the 400–700 nm wavelength range at 10 nm intervals. The XYZ values are calculated by integrating the diffuse reflectance spectra, obtained by the MCS,

(6)

FIGURE 4. In vivo imaging system of the rat fundus.

FIGURE 5. Relative intensity of white LED light source, spectral response of color filters (red, green, and blue), and the molar extinction coefficient spectra of oxygenated hemoglobin and deoxygenated hemoglobin (from Ref. [61]).

using Eq. (2). Then, the transformation matrix N₂is determined using the MRA. After the XYZ values are transformed from the measured RGB values using the first matrix N₁, the XYZ values are transformed into the chromophore concentra- tionsCHbOandCHbRusing matrix N2. The total hemoglobin

FIGURE 6. Fixing method of rat’s front teeth and ears.

FIGURE 7. Control of fraction of inspired oxygen (FiO₂).

concentrationCHbTand oxygen saturation SO2are then calculated asCHbT=CHbO+CHbRand SO₂=(C_HbO/CHbT)× 100, respectively. The absorption coefficients of blood when CHbT=100% are set to those of blood with 44% hematocrit and 150 g/L of hemoglobin [59].

III. MATERIALS AND METHOD

Fig. 4 shows an experimental apparatus that consists of a white LED light source (Fiber-Lite Mi-LED A2, Dolan- Jenner industries Inc., MA, USA), a light guide, a beam splitter (CM1-BS013, Thorlabs Inc., NJ, USA), a zoom lens (VZM 600i, 1.0X-6.0X, Edmund Optics Inc., NJ, USA), a 24-bit RGB CCD camera (DFK23U618, Imaging Source LLC, NC, USA), which has a diagonal 4.5 mm interline CCD solid state image sensor (ICX618AQA, SONY, Tokyo, Japan), and a PC. Figure 5 shows the relative intensity of white LED light source, both the relative response of the red, green, and blue filters mounted on the CCD chips and the molar extinction coefficient of hemoglobin. These color filters are arranged in the Bayer array. An IR cut filter in the sensor rejects unnecessary light greater than 700 nm.

Male Wistar albino rats, which have no melanin and xanthophyll [54]–[56] in their fundus, were used as experimental samples, whereas the human fundus has various pigments,

(7)

FIGURE 8. Imaging of oxygenated hemoglobinC_HbO, deoxygenated hemoglobinC_HbR, total hemoglobin C_HbT, and oxygen saturation SO₂in rat fundus for various fraction of inspired oxygen (FiO₂) values (sample 1). The regions of interest (ROI) represent the optic disk (white frame) and the

tissue area (black frame), respectively.

such as melanin and xanthophyll and the importance of other pigments has been shown for the estimated accuracy of SO2 [4], [38]. In this system, after white light from the light source illuminates the rat eye through the light guide and the beam splitter, the RGB color detector measures both specularly reflected light from the surface of the eye and diffusely reflected light from the fundus that passes through the beam splitter and the zoom lens. The image size was 640×480 pixels. The image was saved as a bitmap (BMP) image. A mydriatic agent (Mydrin-P ophthalmic solution, Santen Pharmaceutical Co., Ltd., Osaka, Japan) was used to dilate the left pupil. In addition, high viscosity eye drops (New Rohto Dry Aid EX Eye Drop, ROHTO Pharmaceutical

Co., Ltd., Osaka, Japan) were used to prevent the eye from drying. The drops also function as a coupling gel, and the eye was covered with a cover glass. A standard white diffuser with 99% reflectance (USRS-99-010, Labsphere Inc., NH, USA) was used to correct the spatial nonuniformity of illumination.

The image of the white diffuser is acquired by the detector after the white diffuser is fixed by a flexible stand (MB-MX, SIGMA KOKI, Tokyo, Japan) at the same position as the eye.

Animal care and experimental procedures were approved by the Animal Research Committee of Tokyo University of Agriculture and Technology. In preparation for the experiment, the rats were put under anesthesia by inhalation of mixed isoflurane and air. Isoflurane was maintained at a

(8)

FIGURE 9. Imaging of oxygenated hemoglobinC_HbO, deoxygenated hemoglobinC_HbR, total hemoglobin C_HbT, and oxygen saturation SO₂in rat fundus for various fraction of inspired oxygen (FiO₂) values (sample 2). The regions of interest (ROI) represent the vein area (blue frame) and the artery area (yellow frame), respectively.

concentration of 2.0% during the experiment. Moreover, to prevent a rat’s head from moving, the rats’ front teeth and ears were fixed using the fixing tool, as Fig. 6 shows. The rat’s mouth and nose were covered with a mask that was connected to an inhalation tube. The isoflurane-air mixture and N₂-O₂mixed gas were pumped into the inhalation tube.

To control the FiO2(fraction of inspired oxygen), the mixture of nitrogen and oxygen was controlled with a gas flowmeter (RK1200M, KOFLOC, Kyoto, Japan) and a valve, while the O2 concentration was monitored with an oxygen monitor- ing system (OM-25MS10, TAIEI Electric Co., Ltd, Tokyo, Japan).

As shown in Fig. 7, we gradually varied the FiO₂by adjust- ing the gas flowmeter and valve for 21 min. The x-axis and

y-axis represent time and FiO₂, respectively. The percentage of O₂ and N₂ contained in the air at 0 m above sea level is 20.9% and 78.1%, respectively. Therefore, an FiO₂value of 21% is similar to normal conditions. The RGB images of the rat’s fundus were captured after the rats breathed for 3 min at each FiO₂step.

IV. RESULTS AND DISCUSSION

Fig. 8 and 9 show the RGB fundus images of two rats (sample 1 and sample 2) and the changes in the images of CHbO,CHbR,CHbT, and oxygen saturation (SO₂), depending on the changes in FiO₂. To make the RGB images shown in Fig. 8 and 9 easy to examine, the contrast and brightness of the images were adjusted. The regions of interest (ROI)

(9)

FIGURE 10. Change in the meanC_HbO,C_HbR,C_HbT, and SO₂within ROI depending on changing fraction of inspired oxygen (FiO₂). Each value in both the optic disc and the tissue area represents the mean value calculated from the ROI shown in Fig. 8 (sample 1). Each value in both the artery and the vein represents the mean value calculated from five ROIs shown in Fig. 9 (sample 2). The error bars show standard deviations.

shown in Fig. 8 represent the optic disk (white frame) and the tissue area (black frame), respectively. Moreover, the ROI shown in Fig. 9 represents the vein area (blue frame) and the artery area (yellow frame). Fig. 10 shows the change in the mean C_HbO,C_HbR,C_HbT, and SO₂ values within the ROI, depending on the change in the FiO₂value. The x-axis denotes FiO₂and the y-axis represents the meanC_HbO,C_HbR, C_HbT_,and SO₂values within the ROI. The breath of the rats was stopped when FiO₂value was 0%.

First, as Fig. 10 shows, C_HbOand SO₂decreased as FiO₂ decreased, whereas the values of C_HbR and C_HbT increased as FiO2decreased. These results correspond to the following physiological phenomena. Decreased partial oxygen pressure (PO2) resulting from decreased FiO2causes defective oxygenation of the blood. Consequently, CHbO decreases and CHbR increases in the artery. This results in decreased SO2

in the artery. Moreover, the low concentration of oxygen in the blood is consumed in the peripheral tissue, and C_HbR shows a greater increase in the peripheral vein. This results in decreased SO₂ in the vein. SO₂ values in capillaries in the optic disk and the tissue area also decreased similarly.

In comparison to the estimated SO₂value shown in Fig. 10

between the artery area and the vein area, the SO2level in the artery area was higher than in the vein and, therefore, our method could distinguish between an artery and a vein.

Next, in Fig. 10,CHbTlargely increased in a hypoxic state at less than 21% FiO₂. This phenomenon shows that defective blood oxygenation is compensated by either vasodila- tion or increased heartbeat. Then,C_HbOand SO₂increased in the samples when the FiO₂was 12%. This occurred presum- ably because the involuntary motion of the eyeball under the anesthesia created variations in the photographic conditions.

Moreover, the SO₂decreased largely when FiO₂ decreased from 14% to 0%. The results indicate that the SO2decreases sharply as oxygen partial pressure becomes close to zero.

To analyze the temporal change in the SO2 in detail, it is necessary to observe the change in fundus color at longer interval with high temporal resolution. Therefore, in future work, we will observe the temporal change in the fundus color by capturing the video.

The estimated SO₂in the tissue area was lower than in the vein. However, the SO₂ in the tissue area should be higher than the vein because the tissue area includes both arteries and veins. This result shows that our fundus model does not

(10)

match the actual fundus completely. For example, the fundus is not flat but a curved surface, whereas the white diffuser that was used for the calibration of our method had a flat surface.

As mentioned before, to correct illuminance non-uniformity, the fundus images were normalized by the image of the white diffuser placed at the same position as the eye. As it goes away from the center of the image, the light intensity of a flat white diffuser is different than for the curved surface of the fundus.

In this sense, the SO₂in the optic disc was reasonable because it was located around the center of the fundus image.

The human fundus has various pigments, such as melanin and xanthophyll [54]–[56] and the importance of other pigments has been shown for the estimated accuracy of SO2 [4], [38]. Therefore, this issue should be improved by proposing a fundus model that has these pigments in future work. Additionally, the model parameters of the MCS simulation rely on the assumptions made and are different from actual biological tissue. Therefore, our simulation will require a calibration using a phantom of the eye or the eyeball of an animal in future work.

Furthermore, our system measures not only diffusely reflected light from the fundus but also specularly reflected light from the surface of the eye. Thus, the saturated pixels shown in Fig. 9 indicate the specular reflection. Particularly for the case when the FiO2was 14%, the estimated values of artery and vein of sample 2 were affected by the specular reflection. In our future work, we plan to reduce the reflection by aligning two polarization plates in the crossed Nicols configuration. Moreover, we will compare the estimation accuracy of SO₂between our method and other methods that use multiple wavelengths.

V. CONCLUSION

We proposed an RGB-based oximetry for the retina. First, we constructed a measuring system for a rat fundus and estimated the SO₂in the rat fundus from the RGB image of the fundus. In addition, we investigated the change inC_HbO, C_HbR,C_HbT, and SO₂by controlling the FiO₂and observing the physiological response caused by the defective oxygenation of the blood. Therefore, because these estimated results could be explained by physiological phenomena, the results indicate the possibility of oximetry based on the RGB images of a fundus. However, our method needs to correct for the influence of both the spatial non-uniformity of illumination by the curved surface of the fundus. In future studies, we will construct a multilayer model, of the ocular fundus, containing melanin and yellow coloring matter (xanthophyll) [54]–[56].

Moreover, we will estimate the SO₂ using the multilayer model and investigate its influence on the estimation of SO₂ across a range of tissue chromophores. In addition, we will compare the results from the single-layer and multilayer models.

REFERENCES

[1] R. D. Jager, W. F. Mieler, and J. M. Miller, ‘‘Age-related macular degeneration,’’New England J. Med., vol. 358, no. 24, pp. 2606–2617, Jun. 2008.

[2] R. Kleinet al., ‘‘The Wisconsin epidemiologic study of diabetic retinopathy: II. Prevalence and risk of diabetic retinopathy when age at diagnosis is less than 30 years,’’Arch. Ophthalmol., vol. 102, no. 4, pp. 520–526, Apr. 1984.

[3] R. Kleinet al., ‘‘The Wisconsin epidemiologic study of diabetic retinopathy: III. Prevalence and risk of diabetic retinopathy when age at diagnosis is 30 or more years,’’Arch Ophthalmol., vol. 102, no. 4, pp. 527–532, Apr. 1984.

[4] D. T. Hartong, E. L. Berson, and T. P. Dryja, ‘‘Retinitis pigmentosa,’’

Lancet, vol. 368, no. 9549, pp. 1795–1809, Nov. 2006.

[5] A. Mirshahi, N. Feltgen, L. L. Hansen, and L.-O. Hattenbach, ‘‘Retinal vascular occlusions: An interdisciplinary challenge,’’Deutsches Ärzteblatt Int., vol. 105, no. 26, pp. 474–479, Jun. 2008.

[6] M. Bhargava, M. K. Ikram, and T. Y. Wong, ‘‘How does hypertension affect your eyes?’’J. Hum. Hypertension, vol. 26, pp. 71–83, Apr. 2011.

[7] C. W. Wong, T. Y. Wong, C.-Y. Cheng, and C. Sabanayagam, ‘‘Kidney and eye diseases: Common risk factors, etiological mechanisms, and path- ways,’’Kidney Int., vol. 85, no. 6, pp. 1290–1302, Jun. 2014.

[8] F. C. Delori and K. P. Pflibsen, ‘‘Spectral reflectance of the human ocular fundus,’’Appl. Opt., vol. 28, no. 6, pp. 1061–1077, Mar. 1989.

[9] P. Fältet al., ‘‘Extending diabetic retinopathy imaging from color to spectra,’’ inProc. SCIA, Oslo, Norway, 2009, pp. 149–158.

[10] P. Bartczaket al., ‘‘Spectrally optimal illuminations for diabetic retinopathy detection in retinal imaging,’’Opt. Rev., vol. 24, no. 2, pp. 105–116, Apr. 2017.

[11] S. S. Hayreh, ‘‘Recent advances in fluorescein fundus angiography,’’Brit.

J. Ophthal., vol. 58, no. 4, pp. 391–412, Apr. 1974.

[12] National Center for Biotechnology Information.PubChem Compound Database; CID=16850. Accessed: Dec. 18, 2018. [Online]. Available:

https://pubchem.ncbi.nlm.nih.gov/compound/16850

[13] National Center for Biotechnology Information.PubChem Compound Database; CID=5282412. Accessed: Dec. 18, 2018. [Online]. Available:

https://pubchem.ncbi.nlm.nih.gov/compound/5282412

[14] R. W. Flower, ‘‘Injection technique for indocyanine Green and sodium fluorescein dye angiography of the eye,’’Invest. Ophthalmol. Vis. Sci., vol. 12, no. 12, pp. 881–895, Dec. 1973.

[15] B. L. Krock, N. Skuli, and M. C. Simon, ‘‘Hypoxia-induced angiogenesis:

Good and evil,’’Genes Cancer, vol. 2, no. 12, pp. 1117–1133, 2011.

[16] S. H. Hardason and E. Stefansson, ‘‘Oxygen saturation in central retinal vein occlusion,’’Amer. J. Ophthalmol., vol. 150, no. 6, pp. 871–875, Dec. 2010.

[17] J. B. Hickam, H. O. Sieker, and R. Frayser, ‘‘Studies of retinal circulation and A-V oxygen difference in man,’’ Trans. Amer.

Clin. Climatol. Assoc., vol. 71, pp. 34–44, 1959. [Online]. Available:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2248999/

[18] J. B. Hickam, R. Frayser, and J. C. Ross, ‘‘A study of retinal venous blood oxygen saturation in human subjects by photographic means,’’Circulation, vol. 27, no. 3, pp. 375–385, Mar. 1963.

[19] J. B. Hickam and R. Frayser, ‘‘Studies of the retinal circulation in man:

Observations on vessel diameter, arteriovenous oxygen difference, and mean circulation time,’’Circulation, vol. 33, no. 2, pp. 302–316, Feb. 1966.

[20] J. M. Beach, K. J. Schwenzer, S. Srinivas, D. Kim, and J. S. Tiedeman,

‘‘Oximetry of retinal vessels by dual-wavelength imaging: Calibration and influence of pigmentation,’’J. Appl. Physiol., vol. 86, no. 2, pp. 748–758, Feb. 1999.

[21] M. Hammer, W. Vilser, T. Riemer, and D. Schweitzer, ‘‘Retinal vessel oximetry-calibration, compensation for vessel diameter and fundus pigmentation, and reproducibility,’’J. Biomed. Opt., vol. 13, no. 5, Sep. 2008, Art. no. 054015.

[22] S. H. Hardarson and E. Stefánsson, ‘‘Retinal oxygen saturation is altered in diabetic retinopathy,’’Brit. J. Ophthalmol., vol. 96, no. 4, pp. 560–563, Mar. 2012.

[23] J. Beach, ‘‘Pathway to retinal oximetry,’’Trans. Vis. Sci. Technol., vol. 3, no. 5, p. 2, Sep. 2014.

[24] D. J. Mordant, I. Al-Abboud, G. Muyo, A. Gorman, A. R. Harvey, and A. I. McNaught, ‘‘Oxygen saturation measurements of the retinal vasculature in treated asymmetrical primary open-angle glaucoma using hyperspectral imaging,’’Eye, vol. 28, no. 10, pp. 1190–1200, Jul. 2014.

[25] H. Arimoto and H. Furukawa, ‘‘Retinal oximetry with 510–600 nm light based on partial least-squares regression technique,’’Jpn. J. Appl. Phys., vol. 49, no. 11R, Nov. 2010, Art. no. 112401.

[26] S. R. Patel, J. G. Flanagan, A. M. Shahidi, J.-P. Sylvestre, and C. Hudson,

‘‘A prototype hyperspectral system with a tunable laser source for retinal vessel imaging,’’Invest. Ophthalmol. Vis. Sci., vol. 54, no. 8, pp. 5163–5168, Aug. 2013.

(11)

[27] K. R. Denninghoffet al., ‘‘Retinal large vessel oxygen saturations correlate with early blood loss and hypoxia in anesthetized swine,’’J. Trauma Acute Care Surg., vol. 43, no. 1, pp. 29–34, Jul. 1997.

[28] K. R. Denninghoff, M. H. Smith, L. W. Hillman, D. Redden, and L. W.

Rue, ‘‘Retinal venous oxygen saturation correlates with blood volume,’’

Acad. Emergency Med., vol. 5, no. 6, pp. 577–582, Jun. 1998.

[29] J. J. Dreweset al., ‘‘Instrument for the measurement of retinal vessel oxygen saturation,’’Proc. SPIE, vol. 3591, pp. 114–120, Jun. 1999. [Online].

Available: https://www.spiedigitallibrary.org/conference-proceedings-of -spie/3591/1/Instrument-for-the-measurement-of-retinal-vessel-oxygen- saturation/10.1117/12.350607.short

[30] M. H. Smith et al., ‘‘Effect of multiple light paths on retinal vessel oximetry,’’Appl. Opt., vol. 39, no. 7, pp. 1183–1193, Mar. 2000.

[31] R. A. Ashman, F. Reinholz, and R. H. Eikelboom, ‘‘Oximetry with a multiple wavelength SLO,’’Int. Ophthalmal., vol. 23, nos. 4–6, pp. 343–346, 2001.

[32] K. R. Denninghoff, M. H. Smith, A. Lompado, and L. W. Hillman, ‘‘Retinal venous oxygen saturation and cardiac output during controlled hemorrhage and resuscitation,’’J. Appl. Physiol., vol. 94, no. 3, pp. 891–896, Mar. 2003.

[33] B. Khoobehiet al., ‘‘Retinal oxygen saturation evaluation by multi-spectral fundus imaging,’’Proc. SPIE, vol. 6511, Mar. 2007, Art. no. 65110B.

[Online]. Available: https://www.spiedigitallibrary.org/conference -proceedings-of-spie/6511/65110B/Retinal-oxygen-saturation-evaluation -by-multi-spectral-fundus-imaging/10.1117/12.710030.short

[34] I. Alabboundet al., ‘‘New spectral imaging techniques for blood oximetry in the retina,’’Proc. SPIE, vol. 6631, Jun. 2007, Art. no. 6631_22. [Online].

Available: https://www.spiedigitallibrary.org/conference-proceedings-of- spie/6631/66310L/New-spectral-imaging-techniques-for-blood-oximetry- in-the-retina/10.1117/12.728535.short

[35] D. J. Mordantet al., ‘‘Spectral imaging of the retina,’’Eye, vol. 25, no. 3, pp. 309–320, Mar. 2011.

[36] H. Furukawa, H. Arimoto, T. Shirai, S. Ooto, M. Hangai, and N. Yoshimura, ‘‘Oximetry of retinal capillaries by multicomponent analysis,’’Appl. Spectrosc., vol. 66, no. 8, pp. 962–969, Jul. 2012.

[37] K. R. Denninghoff, L. DeLuca, K. Sieluzycka, J. K. Hendryx, T. J. Ririe, and R. A. Chipman, ‘‘Retinal oximeter for the blue-Green oximetry technique,’’J. Biomed. Opt., vol. 16, no. 10, Oct. 2011, Art. no. 107004.

[38] R. M. Pritchard, ‘‘Stabilized images on the retina,’’Sci. Amer., vol. 204, no. 6, pp. 72–78, Jun. 1961.

[39] H. C. Hendargo, Y. Zhao, T. Allenby, and G. M. Palmer, ‘‘Snap-shot multispectral imaging of vascular dynamics in a mouse window-chamber model,’’Opt. Lett., vol. 40, no. 14, pp. 3292–3295, Jul. 2015.

[40] L. E. MacKenzie, T. R. Choudhary, A. I. McNaught, and A. R. Harvey,

‘‘In vivooximetry of human bulbar conjunctival and episcleral microvas- culature using snapshot multispectral imaging,’’Exp. Eye Res., vol. 149, pp. 48–58, Aug. 2016.

[41] J. S. Tiedeman, S. E. Kirk, S. Srinivas, and J. M. Beach, ‘‘Retinal oxygen consumption during hyperglycemia in patients with diabetes without retinopathy,’’Ophthalmology, vol. 105, no. 1, pp. 31–36, Jan. 1998.

[42] S. H. Hardarsonet al., ‘‘Automatic retinal oximetry,’’Invest. Ophthalmol.

Vis. Sci., vol. 47, no. 11, pp. 5011–5016, Nov. 2006.

[43] A. R. Harvey, D. W. Fletcher-Holmes, S. S. Kudesia, and C. Beggan,

‘‘Imaging spectrometry at visible and infrared wavelengths using image replication,’’ Proc. SPIE, vol. 5612, pp. 190–198, Dec. 2004.

[Online]. Available: https://www.spiedigitallibrary.org/conference- proceedings-of-spie/5612/0000/Imaging-spectrometry-at-visible-and- infrared-wavelengths-using-image-replication/10.1117/12.580059.short [44] A. R. Harvey, D. W. Fletcher-Holmes, A. Gorman, K. Altenbach,

J. Arlt, and N. D. Read, ‘‘Spectral imaging in a snapshot,’’ Proc.

SPIE, vol. 5694, pp. 110–119, Mar. 2005. [Online]. Available: https:

//www.spiedigitallibrary.org/conference-proceedings-of-spie/5694/1/

Spectral-imaging-in-a-snapshot/10.1117/12.604609.short

[45] A. Gorman, D. W. Fletcher-Holmes, and A. R. Harvey, ‘‘Generalization of the Lyot filter and its application to snapshot spectral imaging,’’Opt.

Express, vol. 18, no. 6, pp. 5602–5608, Mar. 2010.

[46] L. E. MacKenzie and A. R. Harvey, ‘‘Oximetry using multispectral imaging: Theory and application,’’J. Opt., vol. 20, no. 6, Apr. 2018, Art. no. 063501.

[47] F. Cao, Z. Qiu, H. Li, and P. Lai, ‘‘Photoacoustic imaging in oxygen detection,’’Appl. Sci., vol. 7, no. 12, p. 1262, Dec. 2017.

[48] S. Chen, J. Yi, and H. F. Zhang, ‘‘Measuring oxygen saturation in retinal and choroidal circulations in rats using visible light optical coherence tomography angiography,’’Biomed. Opt. Express, vol. 6, no. 8, pp. 2840–2853, Jul. 2015.

[49] M. A. der Puttenet al., ‘‘A multispectral microscope forin vivooximetry of rat dorsal spinal cord vasculature,’’Physiol. Meas., vol. 38, no. 2, pp. 205–218, Jan. 2017.

[50] B. E. Bayer, ‘‘Color imaging array,’’ U.S. Patent 3 971 065, May. 12, 1976.

[51] J. T. Alanderet al., ‘‘A review of indocyanine Green fluorescent imaging in surgery,’’Int. J. Biomed. Imag., vol. 2012, pp. 1–26, Apr. 2012. [Online].

Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3346977/

[52] T. Smith and J. Guild, ‘‘The C.I.E. colorimetric standards and their use,’’

Trans. Opt. Soc., vol. 33, no. 3, pp. 73–134, Jan. 1931.

[53] A. Koschanet al., ‘‘Color spaces and color distances,’’ inDigital Color Image Processing, vol. 1, 1st ed. Hoboken, NJ, USA: Wiley, 2008, ch. 3, sec. 3, pp. 37–44.

[54] P. E. Kilbride, K. R. Alexander, M. Fishman, and G. A. Fishman, ‘‘Human macular pigment assessed by imaging fundus reflectometry,’’Vis. Res., vol. 29, no. 6, pp. 663–674, Feb. 1989.

[55] M. Hammer, A. Roggan, D. Schweitzer, and G. Müller, ‘‘Optical properties of ocular fundus tissues—Anin vitrostudy using the double-integrating- sphere technique and inverse Monte Carlo simulation,’’Phys. Med. Biol., vol. 40, no. 6, pp. 963–978, Jun. 1995.

[56] M. Hammer, D. Schweitzer, E. Thamm, and A. Kolb, ‘‘Non-invasive measurement of the concentration of melanin, xanthophyll, and hemoglobin in single fundus layers in vivo by fundus reflectometry,’’Int. Ophthalmol., vol. 23, nos. 4–6, pp. 279–289, Jun. 2001.

[57] A. A. Stratonnikov and V. B. Loschenov, ‘‘Evaluation of blood oxygen saturationin vivofrom diffuse reflectance spectra,’’J. Biomed. Opt., vol. 6, no. 4, pp. 457–467, Oct. 2001.

[58] V. Tuchin,Tissue Optics: Light Scattering Methods and Instruments for Medical Diagnostics, vol. 1, 2nd ed. Bellingham, WA, USA: SPIE Press, 2007, ch. 1, sec. 1, pp. 3–22.

[59] S. A. Prahl. (Dec. 15, 1999).Tabulated Molar Extinction Coefficient for Hemoglobin in Water. Accessed: Nov. 28, 2018. [Online]. Available:

https://omlc.org/spectra/hemoglobin/summary.html

[60] I. Nishidate, K. Sasaoka, T. Yuasa, K. Niizeki, T. Maeda, and Y. Aizu,

‘‘Visualizing of skin chromophore concentrations by use of RGB images,’’

Opt. Lett., vol. 33, no. 19, pp. 2263–2265, Oct. 2008.

[61] J. R. Mourant, J. P. Freyer, A. H. Hielscher, A. A. Eick, D. Shen, and T. M. Johnson, ‘‘Mechanisms of light scattering from biological cells relevant to noninvasive optical-tissue diagnostics,’’Appl. Opt., vol. 37, no. 16, pp. 3586–3593, Jul. 1998.

[62] D. Abookasis, C. C. Lay, M. S. Mathews, M. E. Linskey, R. D. Frostig, and B. J. Tromberg, ‘‘Imaging cortical absorption, scattering, and hemodynamic response during ischemic stroke using spatially modulated near- infrared illumination,’’J. Biomed. Opt., vol. 14, no. 2, Mar./Apr. 2009, Art. no. 024033.

[63] A. Mustariet al., ‘‘In vivoevaluation of cerebral hemodynamics and tissue morphology in rats during changing fraction of inspired oxygen based on spectrocolorimetric imaging technique,’’Int. J. Mol. Sci., vol. 19, no. 2, pp. 491-1–491-15, Feb. 2018.

[64] A. Mustari, N. Nakamura, S. Kawauchi, S. Sato, M. Sato, and I. Nishidate, ‘‘RGB camera-based imaging of cerebral tissue oxygen saturation, hemoglobin concentration, and hemodynamic spontaneous low- frequency oscillations in rat brain following induction of cortical spreading depression,’’Biomed. Opt. Express, vol. 9, no. 3, pp. 933–951, Mar. 2018.

[65] F. P. Bolin, L. E. Preuss, R. C. Taylor, and R. J. Ference, ‘‘Refractive index of some mammalian tissues using a fiber optic cladding method,’’Appl.

Opt., vol. 28, no. 12, pp. 2297–2303, Jun. 1989.

KAZUYA NAKANOreceived the M.S. and Ph.D.

degrees from the Tokyo Institute of Technology, in 2007 and 2013, respectively. He was with the Mizuho Information and Research Institute (MHIR), from 2007 to 2009. He was a Specially Appointed Assistant Professor with Nippon Sport Science University, from 2013 to 2014, and the Tokyo University of Science, from 2015 to 2016.

Since 2017, he has been a Specially Appointed Assistant Professor with the Center for Fron- tier Medical Engineering, Chiba University. His current research interests include diffuse reflectance spectroscopy, optical security, information pho- tonics, and applied physics.

(12)

RYOSUKE HIROFUJIreceived the B.S. and M.S.

degrees in engineering from the Tokyo University of Agriculture and Technology, in 2015 and 2017, respectively. He is currently with NIDEC CO., LTD. His current research interest includes eye care.

TAKASHI OHNISHIreceived the M.S. and Ph.D.

degrees from Chiba University, in 2010 and 2013, respectively. He is currently an Assistant Professor with the Center for Frontier Medical Engineering, Chiba University. His current research interests include image processing, image registration, and pathological images.

MARKKU HAUTA-KASARI received the M.S.

degree in computer science from the University of Kuopio, Finland, in 1994, and the Ph.D. degree in information processing from the Lappeenranta University of Technology, Finland, in 1999. He is currently a Professor in computer science with the University of Eastern Finland. He is also the Head of the Spectral Color Research Group. His research interests include spectral color research, pattern recognition, and computer vision.

IZUMI NISHIDATEreceived the M.S. and Ph.D.

degrees in mechanical systems engineering from the Muroran Institute of Technology, in 2001 and 2004, respectively. In 2004, he joined the Depart- ment of Bio-Systems Engineering, Yamagata Uni- versity, as an Assistant Professor. In 2007, he joined the Graduate School of Bio-Applications and Systems Engineering, Tokyo University of Agriculture and Technology, as an Associate Pro- fessor. He is currently an Associate Professor with the Graduate School of Bio-Applications and Systems Engineering, Tokyo University of Agriculture and Technology. His major research interests include diffuse reflectance spectroscopy, spectral imaging, analysis of light transport in biological tissues, and functional imaging of various organs.

HIDEAKI HANEISHIreceived the M.S. and Ph.D.

degrees from the Tokyo Institute of Technology, in 1987 and 1990, respectively. He joined Chiba University as a Research Associate, in 1990. He was a Visiting Research Scientist with the Depart- ment of Radiology, University of Arizona, from 1995 to1996. He has been a Full Professor of the Center for Frontier Medical Engineering (CFME), since 2007. He is also the Director of CFME.