3D shape measurement based on structured light field imaging

1.   Introduction

Light field imaging (LFI) has been increasingly popular in the last decade, which records the angular and spatial distribution of rays in all directions in the space simultaneously. The captured light field enables the 3D shape measurement, which is applied to several areas, such as the robotics, automation, medicine and so on ^[1]. Among these applications, an area with great potential for LFI is biomedical imaging. Basing on the conventional microscope, Levoy et al. ^[2] inserted a micro-lens array into the optical train and realized the reconstruction volumes for various biological specimens by focal stacks. Furthermore, Yong ^[3] developed the light field microscopy and enabled the observation of the 3D surface morphology of opaque microstructures. Liu et al. ^[4] constructed a prototype light field endoscopy for invasive surgery.

Light field imaging is conventionally achieved with dense camera array ^[5] before the lens-let light field camera (LFC) appeared ^[6]. In 2005, Ren Ng accomplished a hand-held lens-let LFC by introducing a micro lens array (MLA) into the image plane of a conventional camera ^[7]. The angular information recorded by LFC allows the depth estimation or scene reconstruction with or without calibration results ^[8]. Ng et al. ^[7] proposed that a light field image can be refocused in various depth from sub-aperture images based on Fourier Slice Theorem. What’s more, Tao ^[9] combines correspondence cues, defocus cues and shading cues of sub-aperture images to accomplish the depth estimation. In addition, the slopes of lines in epipolar images (EPI) indicate the corresponding depths of the object, which is often used to achieve depth estimation. Criminisi ^[10] proposed an EPI volume and found high degree of regularity to compute the depth. Kim ^[11] utilized high angular resolution EPI and performed confidence measurement to assess the reliability of the depth. Wanner and Goldluecke ^[12] performed structure tensor operator on EPI to calculate the slopes, and Diebold ^[13] extended structure tensor to the field of heterogeneous light fields to deal with non-constant intensity problem in EPI. However, due to noise, low angular resolution and textureless regions, the methods mentioned above only generate coarse initial 3D shape information when calibration is performed, and the furthermore refinement has to be executed.

The basic idea to refine the 3D coordinates in this paper is inspired by Markov Random Field (MRF). Tao ^[9] applied the MRF framework to initial depth, and Wanner and Goldluecke ^[12] took the depth reconstruction as a multi-label problem and employed a total variance smoothing model to achieve the optimization. The MRF framework proposed by Tao ^[9] is based on the smoothness and color consistency, which are taken as extra constraints to optimize the final depth. Structured light field makes it possible to establish more constraints. Structured light field means projecting a pattern onto an object, the deformed pattern modulated by the object surface encodes 3D shape information and is acquired by the light field camera, which enables the 3D shape measurement. For a certain point, the 3D coordinates are encoded not only in LFC sub-aperture images but also in the modulated grating pattern, which provides additional constraints to accomplish the 3D shape measurement accurately.

In this paper, a structured light field system is established, which consists of a lens-let LFC and a projector, as shown in Figure 1. The remainder of this paper starts with system calibration, including the light field camera calibration and system calibration. The initial 3D coordinates are derived based on phase consistency in EPI and LFC calibration results. Subsequently, the 3D coordinates are refined based on the relationship between the structured light field image and DMD image of the projector. As a Consequence, the experimental results are presented to demonstrate the 3D shape measurement performance.

Figure 1.  The structured light field system.

DownLoad: Full-Size Img PowerPoint

2.   Structured light field system calibration

The structured light field system in this paper consists of a lens-let LFC and a projector. Therefore, the system calibration includes the LFC calibration, projector calibration and the system joint calibration. The model of the whole SLF system is described in Figure 2(b) and the coordinate system settings are listed in Table 1.

Figure 2.  (a) Mathematical model of a lens-let LFC. (b) The model and coordinates configuration of the SLF system.

DownLoad: Full-Size Img PowerPoint

Table 1.  Coordinate system settings of SLF system description.

Coordinate System	Description
$\left( {{O^w}, {X^w}, {Y^w}, {Z^w}} \right)$	World coordinate system
$\left( {{O^c}, {X^c}, {Y^c}, {Z^c}} \right)$	Light field camera coordinate system
$\left( {o_n^c, x_n^c, y_n^c} \right)$	Sub-aperture image coordinate system (pixel coordinate)
$\left( {{O^m}, {X^m}, {Y^m}} \right)$	Sub-aperture image coordinate system (physical coordinate)
$\left( {{O_{ij}}, i, j} \right)$	CCD coordinate system
$\left( {{O_{uv}}, u, v} \right)$	Element image coordinate system
$\left( {{O^p}, {X^p}, {Y^p}, {Z^p}} \right)$	Projector coordinate system
$\left( {o_n^p, x_n^p, y_n^p} \right)$	DMD image coordinate system (pixel coordinate)

---

 | Show Table

DownLoad: CSV

2.1.   Light field camera calibration

The mathematical model of a typical lens-let LFC is shown in Figure 2(a). To clarify the recorded information, $L\left( {s, t, x, y} \right)$ is simplified to 2D $L\left( {s, x} \right)$ in Figure 2(a). $L\left( {s, x} \right)$ expresses a ray that passes through the main-lens plane and MLA plane. In Figure 2(a), $s$ is the distance from optical center ${O_C}$ to the sub-region of the main-lens where the rays passes through the main-lens plane, ${x^m}$ is the distance from the MLA’s center to the position where the ray passes through the MLA plane.

The projection from a scene point $P\left( {{x^c}, {z^c}} \right)$ to the MLA plane can be expressed as:

where ${h'_m}$ is the distance from the main-lens plane to the MLA plane, ${h_m}$ is the focused depth of the main-lens, and $d$ is the micro-lens diameter. The coordinates on the MLA plane and image plane are shown in Figure 2(b).

In general, each sub-aperture image is the result of taking the same pixel underneath each micro-lens, at the offset corresponding to $\left( {u, v} \right)$ for the sub-region of the main-lens aperture. Therefore, the distance between two adjacent sub-apertures can be expressed as $D = {{q{{h'}_m}} \mathord{\left/ {\vphantom {{q{{h'}_m}} b}} \right. } b}$, where $b$ is the distance from the MLA plane to the image plane, as shown in Figure 2(a).

The projection relationship between $s$ and $u$ is described as:

where $q$ is the pixel size of image sensor. Moreover, the relationship between two coordinates in MLA plane is expressed as:

Let ${s_k} = D\left( {{u_k} - {u_0}} \right)$ and ${s_{k + 1}} = D\left( {{u_{k + 1}} - {u_0}} \right)$ describe two adjacent sub-apertures so that Eq (1) can be rewritten ^[14] as

where $\Delta x$ is the disparity value in disparity map, constructed with two adjacent sub-aperture images. Theoretically, the central sub-aperture image is the same as a picture that captured by a conventional camera with a same aperture. Therefore, the main-lens parameters of the lens-let LFC can be calibrated based on Zhang’s calibration method ^[15] using central sub-aperture images taken in different viewpoints.

Except for the parameters of the main-lens, the other parameters in Figure 2(a) need to be calibrated also. To calibrate the Eq (4), which shows the constraint between the depth ${z^c}$ and disparity $\Delta x$, ${z^c}$ of each feature point in LFC images is calculated using the main-lens parameters, and $\Delta x$ is acquired through the corresponding EPI. Therefore, the parameters $b$, $D$ and ${h_m}$ are derived with the line-fitting coefficients in Eq (4). The light field camera calibration is described in detail in our previous work ^[16].

The commercial light field camera, Lytro Illum is equipped in the structured light filed based 3D shape measurement system. Its pixel size $q$ is 0.0014 mm and the micro-lens diameter $d$ is 0.02 mm. The light field images are decoded with the toolbox provided by Donald ^[17], the resolution of sub-aperture images is ${\rm{434}} \times {\rm{625}}$ pixels. The calibrated parameters of Lytro Illum are as follows: $f$ is 40.11 mm, ${h'_m}$ is 40.88 mm, ${h_m}$ is 2116.34 mm, $b$ is 48.38 um, $D$ is 1.18 mm, and the center of MLA plane, $\left( {{x_0}, {y_0}} \right)$ is $\left( {312.67,225.53} \right)$. The average re-projection error is 0.0685 pixels. Especially, Eq (4) is fitted as

2.2.   System calibration

To accomplish 3D shape measurement, the structured light field system in this paper is calibrated also. The calibration methods are similar to that of the conventional structured light system equipped with a projector and a conventional camera. In our calibrations, the central sub-aperture image is utilized instead of conventional camera images.

For the sake of completeness, the projector calibration and conventional structured light system calibration are presented briefly which have been reported previously in detail ^[18]. To calibrate the projector, the projector’s digital micromirror device (DMD) image is needed. The 4-step phase-shifting algorithm and heterodyne principle are applied to obtain the unwrapped phase map in both horizontal and vertical directions. More details on the phase-shifting and heterodyne are described in ^[19]. Although the fringe patterns are captured by the light field camera, the unwrapped phase map in the central sub-aperture image is calculated only in the system calibration. Therefore, the DMD image is formed in such a way that a univocal correspondence is established between the unwrapped phase in the central sub-aperture image and the DMD image in horizontal and vertical directions.

To calibrate structured light field system, the conventional stereo calibration method is applied, where the central sub-aperture image and DMD image are used. The intrinsic and extrinsic parameters of the light field camera and projector are computed, so are the transform matrices between them. Eight groups of central sub-aperture images in different views are captured and applied in the calibration, and the calibration results are shown in Figure 3, where the light field camera and the projector are labeled as LFC and Pro, respectively. The LFC is modelled as Figure 2(a) and the projector is treated as an inverse camera ^[18]. The reason why the projector is closer to the calibration board is matching the field of view to that of the light field camera. The average re-projection error is 0.2189 pixels.

Figure 3.  The structured light field system calibration results.

DownLoad: Full-Size Img PowerPoint

3.   3D shape measurement methods

It is essential for the 3D shape measurement based on structured light field to retrieve accurate coordinates. To improve the real-time capability, only two sinusoidal gratings with π shift are projected onto the object in this paper, and the deformed pattern is captured by the LFC. The depth of the object is derived from the phase-consistency-based slope estimation method in EPI according to Eq (5), and the 3D coordinates of the object are derived from the LFC calibration results subsequently. Furthermore, the 3D coordinates refinement is performed to accomplish the 3D shape measurement accurately.

3.1.   EPI-based depth computation

The light field is denoted $L\left( {s, t, x, y} \right)$, where $\left( {s, t} \right)$ is the sub-aperture position in main-lens plane and $\left( {x, y} \right)$ is the micro-lens position in MLA plane. Each epipolar image (EPI) is the 2D slice of the light field image where $\left( {t, y} \right)$ are fixed, and $\left( {s, x} \right)$ vary, and vice versa. An EPI-based depth computation approach is proposed in this paper, as illustrated in Eq (5). In EPI, the disparity is inversely proportional to the line’s slope for a certain point in the scene ^[14], that is $\Delta x = {1 \mathord{\left/ {\vphantom {1 k}} \right. } k}$ where $\Delta x$ is the disparity for the point in the scene, and $k$ is the corresponding line’s slope in the EPI. Therefore, the depth of a certain point is derived from Eq (5).

To obtain the slope $k$, instead of previous radiance variance methods ^[20], the phase variance method is proposed in this paper. Generally, the sub-aperture images are treated as observing the object from different viewpoints. Based on the phase measurement profilometry (PMP) principles, the phases in different sub-aperture images modified by the same object point are almost the same, so that the phase information shows consistency along the corresponding line in the EPI, which is termed phase consistency in this paper. Therefore, the EPI used for slope computation based on phase consistency is not the 2D slice of original light field, but that of the structured light field after phase computation in all of the sub-aperture images.

For a certain sub-aperture image, the captured fringe pattern image is expressed as:

where $I\left( {x, y} \right)$ is the image intensity, $A\left( {x, y} \right)$ is the background intensity, $B\left( {x, y} \right)$ is the modulated intensity of a given pixel and $\phi$ is the desired phase information modulated by the object depth and to be solved. $\phi$ varies in the range of $0 \sim 2\pi$ and thus termed wrapped phase. $A\left( {x, y} \right)$ is the background intensity, which is assumed to be constant but actually change with the environment. So that the conventional Fourier transform profilometry method performs badly when the frequency bandwidth of the $A\left( {x, y} \right)$ is too wide to be separated. Therefore, the modified Fourier transform profilometry method is applied in this paper. Another fringe pattern image ${I_{\rm{1}}}\left( {x, y} \right)$ with $\pi$ shift based on $I\left( {x, y} \right)$ is captured:

Therefore, the wanted $\phi$ can be derived by applying the Hilbert Transform to the difference between $I\left( {x, y} \right)$ and ${I_{\rm{1}}}\left( {x, y} \right)$. And then the complex logarithm is applied, the imaginary part is desired wrapped phase which varies in the range of $0 \sim 2\pi$, i.e.$\phi = {\rm{im}}\left[ {\log \left( {I + i \cdot H\left( {I - {I_1}} \right)} \right)} \right]$, where $H\left( \bullet \right)$ is the Hilbert Transform operator. The equation is applied to all the sub-aperture images and the EPI is derived at last, where the phase consistency is utilized to compute the slopes of lines.

Without losing generosity, for a pixel $P$ in the EPI, $n$ lines ${l_1}:{l_n}$ are chosen as shown in Figure 4, where $P$ is in the middle of the lines. The corresponding slopes and phase variances of the lines are expressed as ${k_1}:{k_n}$ and ${v_1}:{v_n}$, respectively. Therefore, the optimal line ${l_i}$ for a certain point in the scene is obtained when the phase variance reaches the minimum value, and the corresponding optimal disparity $\Delta x$ is the reciprocal of the optimal slope $k$, i.e. ${k_{\arg \min \left( {{v_i}} \right)}}$. Therefore, for a certain point in the scene, its depth is derived with the optimal disparity according to Eq (5), and the other 3D coordinates are derived with the calibration parameters of the light field camera, as shown in Eq (8).

Figure 4.  Computation of the line’s slope in EPI.

DownLoad: Full-Size Img PowerPoint

3.2.   3D coordinates refinement

With the slope estimation in EPI and LFC calibration results, the 3D coordinates of object are derived. However, the EPI only provides coarse depth measurement result due to the low angular resolution, which results in the inaccuracy of 3D coordinates. In the structured light field imaging system in this paper, the depth is not only indicated by the lines’ slopes in the EPI, but also encoded in the phase carried by the modulated sinusoidal grating, which is relatively accurate and provides significant constraint to refine the initial 3D shape measurement results.

In the SLF system, an object point ${P_W}$ is projected onto two image planes, the LFC image plane and DMD image plane, respectively. Therefore, the phase information recorded in sub-aperture images and DMD image is considered in our method. The wrapped phase in the central sub-aperture image is derived with the methods in Section 3.1. On the other hand, the unwrapped phase is computed by re-projecting the 3D coordinates reconstructed in Section 3.1 to DMD image plane, which can also be turned into wrapped phase by performing the modulo function. Therefore, there are some phase cues to refine the 3D shape measurement results. To facilitate the refinement method, the accurate unwrapped phase of the object point ${P_W}$ is termed ${\phi _{un}}\left( {{P_W}} \right)$. ${P_C}$ is the corresponding point of ${P_W}$ on the central sub-aperture image and ${\phi _{wr}}\left( {{P_C}} \right)$ is the wrapped phase of ${P_C}$, as shown in Figure 5(a). The relationship between the ${\phi _{un}}\left( {{P_W}} \right)$ and ${\phi _{wr}}\left( {{P_C}} \right)$ is expressed as ${\phi _{wr}}\left( {{P_C}} \right) = \bmod \left( {{\phi _{un}}\left( {{P_W}} \right), 2\pi } \right)$. Based on the method proposed in Section 3.1, the 3D coordinates of ${P_W}$ are derived as follows

Figure 5.  (a) The wrapped phase map. (b) The wrapped phase map calculated with re-projection phase map. (c) The illustration of correcting (b) with (a).

DownLoad: Full-Size Img PowerPoint

where ${{\bf{X}}^{\bf{W}}} = {\left( {{x^w}, {y^w}, {z^w}} \right)^T}$ are the 3D coordinates of ${P_W}$ in the world coordinate system, ${{\bf{X}}^{\bf{C}}} = {\left( {{x^c}, {y^c}, {z^c}} \right)^T}$ are the 3D coordinates of ${P_W}$ in the LFC coordinate system, $\left( {x_n^c, y_n^c} \right)$ are the pixel coordinates of ${P_C}$ on central sub-aperture image, $\left( {x_{n0}^c, y_{n0}^c} \right)$ is the central pixel coordinate on central sub-aperture image, ${{\bf{R}}_{\bf{L}}}$ is the rotation matrix of LFC and ${{\bf{t}}_{\bf{L}}}$ is the translation vector of LFC.

As mentioned above, the 3D coordinates of ${P_W}$ are coarse and will be refined, which is accomplished by correcting the coarse unwrapped phase of ${P_W}$. Let ${P_D}$ be the corresponding re-projection point of ${P_W}$ on the DMD image, the coordinates of ${P_D}$ are derived from Eq (10).

where $\left( {x_n^p, y_n^p} \right)$ are the coordinates of ${P_D}$, ${{\bf{M}}_{{\bf{W2P}}}}$ is the transform matrix from world coordinate system to Projector coordinate system, and ${{\bf{M}}_{{\bf{P2D}}}}$ is the transform matrix from Projector coordinate system to DMD image coordinate system. The unwrapped phase of ${P_D}$ is ${\phi _{re}}\left( {{P_D}} \right)$, which is termed re-projection phase and derived from Eq (11)

where $N$ is the number of periods of the projected fringe patterns, $re{s_y}$ is the resolution of projector DMD image in $y$ direction. The corresponding wrapped phase of ${\phi _{re}}\left( {{P_D}} \right)$ is termed ${\phi _{wr}}\left( {{P_D}} \right)$. The relationship between ${\phi _{wr}}\left( {{P_D}} \right)$ and ${\phi _{re}}\left( {{P_D}} \right)$ is similar to that of ${\phi _{wr}}\left( {{P_C}} \right)$ and ${\phi _{un}}\left( {{P_W}} \right)$. When the 3D shape measurement is accurate, ${\phi _{re}}\left( {{P_D}} \right)$ is equal to ${\phi _{un}}\left( {{P_W}} \right)$ and ${\phi _{wr}}\left( {{P_D}} \right)$ is equal to ${\phi _{wr}}\left( {{P_C}} \right)$. However, the re-projection phase ${\phi _{re}}\left( {{P_D}} \right)$ derived from the coarse 3D coordinates of ${P_W}$ is inaccurate due to the error of disparity, so that the unwrapped phase of ${P_W}$ is corrected by the re-projection phase ${\phi _{re}}\left( {{P_D}} \right)$.

The re-projection phase ${\phi _{re}}\left( {{P_D}} \right)$ is corrected according to the relationship between unwrapped phase and wrapped phase, as shown in Figure 5. A point ${P'_D}$ is given as shown in Figure 5(b) and (c), which lies in the same period of ${P_D}$ and whose wrapped phase ${\phi _{wr}}\left( {{{P'}_D}} \right)$ is equal to ${\phi _{wr}}\left( {{P_C}} \right)$. The period of ${P_D}$ is obtained by dividing ${\phi _{re}}\left( {{P_D}} \right)$ with 2π.

When the difference between ${\phi _{wr}}\left( {{{P'}_D}} \right)$ and ${\phi _{wr}}\left( {{P_D}} \right)$ is less than $\pi$, the re-projection phase ${\phi _{re}}\left( {{P_D}} \right)$ of ${P_D}$ is corrected to be as same as that of the point ${P'_D}$. Nevertheless, the re-projection phase is corrected as Eq (12), when the difference is larger than $\pi$, as shown in Figure 5(b) and (c).

Consequently, the 3D coordinates of ${P_W}$ are refined according to the corrected re-projection phase ${\phi _{re}}\left( {{P_D}} \right)$ and ${P_C}$ based on the PMP principles, so the 3D shape measurement is accomplished accurately.

4.   Experiments results and analysis

The light field camera, Lytro Illum is equipped in the 3D shape measurement system, whose analytic image angular resolution is $15 \times 15$ pixels and spatial resolution is $434 \times 625$ pixels. The projector, BENQ GP1, is equipped in the system also, whose resolution is $600 \times 800$ pixels. To evaluate the 3D shape measurement performance, a sinusoidal fringe pattern with 36 periods is projected onto a sculpture surface, and the modulated fringe pattern image is captured by the light field camera, as shown in Figure 6(a).

Figure 6.  (a) The original pattern image of a sculpture. (b) The wrapped phase from central sub-aperture image. (c) The disparity map of the sculpture from EPI. (d) Re-projection phase map from 3D coordinates. (e) Wrapped phase map from (d). (f) Unwrapped phase map after correction. (g) Wrapped phase map from (f).

DownLoad: Full-Size Img PowerPoint

The wrapped phase map in the central sub-aperture image is computed and shown in Figure 6(b), where the background intensity and the modulated intensity have been removed by the modified Fourier transform profilometry ^[21]. The wrapped phase of every pixel in the Figure 6(b) is in the range of 0:2π. Subsequently, the EPI-based method is performed and the disparity map of the sculpture is shown in Figure 6(c), where the phase consistency method is substituted for the radiance consistency to estimate the slopes. Unfortunately, the 3D shape of the sculpture based on disparity map and calibration parameters directly is not accurate enough. Therefore, the 3D coordinates are refined by the phase information mentioned in Section 3.2. The re-projection phase map before and after correction are shown in Figure 6(d) and (f), respectively. To illustrate the re-projection phase map correction performance, the wrapped phase maps of re-projection phase map before and after correction are computed, as shown in Figure 6(e) and (g), respectively. For instance, the phase of the smooth surface of the object is supposed to change smoothly and continuously, as shown in the area (2) of Figure 6(g), which is more accurate than area (1) of Figure 6(e).

The 3D shape of the sculpture is reconstructed with the corrected unwrapped phase map and calibration results, as shown in Figure 7, where the lens distortions of LFC and projector are corrected according to our previous works ^[22].

Figure 7.  The depth of the reconstructed sculpture.

DownLoad: Full-Size Img PowerPoint

To evaluate the 3D shape measurement performance, the experimental result in Figure 7 is compared to that measured by a conventional PMP system. In this paper, the conventional PMP system is equipped with a projector and a light field camera also, while the central sub-aperture image is considered as the image acquired by the conventional camera. Furthermore, the fringe pattern images with periods of 36, 30 and 25 are projected, and the heterodyne principle is applied to achieve the unwrapped phase map, which is considered as the ground truth of the sculpture 3D shape measurement. As the central sub-aperture images are used in both the 3D shape measurement systems, it’s obvious that the measurement error is due to the different unwrapped phase maps completely. The difference maps between the ground truth and unwrapped phase maps after and before correction are computed to analyze the 3D shape measurement performance, as shown in Figure 8.

Figure 8.  (a) The ground truth of the unwrapped phase. The error of the unwrapped phase map (b) after and (c) before correction.

DownLoad: Full-Size Img PowerPoint

As mentioned above, the disparity map derived from EPI-based method is not accurate enough for the 3D shape measurement, so that the re-projection phase map is worse than the final unwrapped phase map after correction. The re-projection phase error relative to the ground truth is shown in Figure 8(b). It’s obvious that the error of the unwrapped phase map after correction is much smaller than that before correction, so that the corrected unwrapped phase refines the 3D coordinates consequently and improves the 3D shape measurement performance. For instance, as shown in the white square frames in Figure 8(b) and (c), the accuracy has been improved remarkably but the error cannot be corrected completely due to the slightly high gradient of depth. However, as only two fringe patterns with different phases are projected onto the sculpture in a specific angle, the loss of viewpoints results in the bad illumination in the transitional region between the right cheek, ear, and neck, as shown in the red square frame in Figure 8(a) (the ground truth) where the phase accuracy decreases heavily due to the low SNR.

5.   Conclusion

In this paper, a 3D shape measurement method based on structured light field is accomplished, where only two sinusoidal fringes are projected onto the object and are acquired by the light field camera. The performance of 3D shape measurement has been improved by the means of utilizing phase information, i.e. phase consistency to derive initial coarse 3D coordinates and unwrapped phase correction to refine the coarse 3D coordinates. Future works will focus on reducing the error caused by high gradient of depth and the biomedical imaging system configuration based on structure light field.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (NSFC) 61771130 and “the Fundamental Research Funds for the Central Universities”.