A model-based method with geometric solutions for gaze correction in eye-tracking

1.   Introduction

Although the past two decades have witnessed the rapid development of eye-tracking technology, there are still some defects for deriving the accurate gaze estimation. Most model-based gaze estimation methods often require prior parameters which are usually set as constants in practice ^[1]. However, recent studies have shown that eyeballs could have ethnic characteristics and individual differences ^[2]. In this condition, the use of constant intrinsic parameters for gaze estimation could lead to deviations by ignoring individual differences of eyeballs. Furthermore, the distortions of eyeball can affect the accuracy of eye-tracking data when using the general model-based method for gaze estimation. Myopia, a globally popular eye disease, can cause the distortion of eyeball, such as the increase in the axial length of eyeball. Strabismus, another common eye disease, can conduct the misalignment of visual axes. Moreover, it has been reported that high myopia is often accompanied by strabismus ^[3]. The hybrid changes of eyeball caused by eye diseases can finally lead to system errors of gaze estimation when using the general eyeball model. Therefore, the correction approach is required in model-based gaze estimation methods, especially for myopia and strabismus which are not rare in the population.

Some studies have been done to introduce the solutions for correcting the gaze estimations and improving the accuracy of eye-tracking data in several applications of eye tracker. A new method named surface recalibration is reported for gaze correction to improve the performance of gaze-based human-computer interaction ^[4]. SPOCK target projections is proposed for online recalibration of gaze data in gaze-only user interaction ^[5]. An algorithm using best-fitting linear transformation is conducted for the offline correction of gaze data in eye-tracking experiments ^[6]. A regression-based approach for mode-of-disparities error correction is applied to correct the gaze data of a visual search experiment ^[7]. A compensation method for head movement is proposed by Zhu ^[8] based on the geometry relationship of the subject's head and plane parameters. Meanwhile, a model-based algorithm in Riemannian space is proposed to compensate the error of gaze estimation in Euclidean geometry eye tracking system in China ^[9].

Nevertheless, most gaze correction algorithms are established by data fitting without considering the systematic errors from the eyeball. In light of the conditions of myopia and strabismus, this paper proposes a algorithm to correct the error in eye-tracking data based on systematic errors of eyeball, which is obtained by calculating the geometric relationship between actual gaze data and the eye-tracking data in the step of calibration when using an eye tracker. Then, computer simulations are used to verify the validity of the proposed model-based algorithm.

2.   The gaze correction method based on eyeball model

In general, eyeball can be regarded as a standard sphere rotating around a fixed point for simplifying the calculation in algorithms, the eyeball center is 13.5 mm behind the cornea, the pupil center is on the spherical surface ^[12], as shown in Figure 1. The line between eyeball center and pupil center is the line-of-sight.

For using an eye tracker, nine-point calibration is a necessary procedure to determine certain person-specific parameters in the eye tracking technology ^[13]. The correction of eye-tracking data can be briefly summarized as the following three steps: (1) obtaining the systematic error of eyeball based on the actual gaze data and the eye-tracking data; (2) calculating data error based on systematic error model of eyeball; (3) correcting the eye-tracking data.

Figure 2 shows the positions of the eye and the screen when using the eye tracker. The center of eyeball model is recorded as O, and the position of the eye tracker is noted as point Z. When facing forward, the intersection of line-of-sight and the screen is noted as A. Usually, the coordinate of A and the length of OZ can be derived by the eye tracker in the eye tracking technology. Obviously, OA $\perp$ AZ, the length OA is a constant if subject's head remains stationary, it can be derived by the two quantities mentioned above. The coordinates of the predefined points are known, and the corresponding estimated coordinates of gaze points can be derived by the eye tracker in the step of calibration.

As mentioned before, individual differences and the distortions that caused by eye diseases such as myopia and strabismus may cause deviation between the eye-tracking data based on the model-based gaze estimation method and the actual gaze data. Myopia causes the subject's eyeball to bulge, myopia eyeball can be assumed to be a sphere model with a larger diameter than the normal eyeball model; strabismus refers to the phenomenon that the optical axis of eyeball is significantly deviated with the visual axis. However, there is only a slight difference between the optical axis and the visual axis in normal eye. These factors of eye diseases can cause difference from the reference model in eye tracking technology. In this study, a single eyeball is taken as an example to analyze the principle of the model-based correction method. It can be easily extended to the binocular system.

2.1.   Gaze correction for strabismus

Firstly, the eyeball model of strabismus is introduced. Generally, strabismus includes horizontal strabismus, vertical strabismus, and torsional strabismus. For a single eye, the deviation between the optical axis and the visual axis is generally considered to remain the same in a certain period. Here we take the torsional strabismus to analyze the principle of correction, while the horizontal strabismus and the vertical strabismus are the special cases of the torsional strabismus.

In the condition of strabismus, the geometric relationship between the actual and estimated gaze points is illustrated in Figure 3, where the point O and A is corresponding to the O and A in Figure 2, while J is the actual gaze point of strabismus at the screen of computer monitor, J$_{\rm 1}$ is the estimated gaze point of J. As shown in Figure 3, OA is perpendicular to the screen plane. Then, the deviation between J and J$_{\rm 1}$ can be decomposed into horizontal and vertical sections in the screen plane, which the point B and B$_{\rm 1}$ are the projections of J and J$_{\rm 1}$ on the x-axis, the point C and C$_{\rm 1}$ are the projections of J and J$_{\rm 1}$ on the y-axis. In the Cartesian coordinate system ${xAy}$ shown in Figure 3, the coordinates of point J, J$_{\rm 1}$, B, B$_{\rm 1}$, C and C$_{\rm 1}$ are ($x_J, y_J$), ($x_{J_1}, y_{J_1}$), ($x_B, y_B$), ($x_{B_1}, y_{B_1}$), ($x_C, y_C$) and ($x_{C_1}, y_{C_1}$). It is obvious that $x_J = x_B$, $y_J = y_C$, $x_{J_1} = x_{B_1}$, $y_{J_1} = y_{C_1}$, and the angle $\angle$AOJ, $\angle$JOJ$_{\rm 1}$, $\angle$AOB, $\angle$BOB$_{\rm 1}$, $\angle$AOC and $\angle$COC$_{\rm 1}$ are noted as $\theta$, $\theta^\prime$, $\alpha$, $\alpha^\prime$, $\beta$ and $\beta^\prime$. According to the principle of eye-tracking and the visual function, it is generally considered that all those angles are limited in the range of $0^\circ$ $\sim$ $45^\circ$. In this study, due to the screen size, the actual range of $\theta$, $\theta^\prime$ is $0^\circ$ to $23.8^\circ$, the range of $\alpha$, $\alpha^\prime$ is $0^\circ$ to $21^\circ$, and the range of $\beta$, $\beta^\prime$ is $0^\circ$ to $12^\circ$. The angle $\angle$JOJ$_{\rm 1}$ ($\theta^\prime$) can be used to describe the deviation angle cased by strabismus, which can be decomposed into a horizontal deviation angle $\angle$BOB$_{\rm 1}$ ($\alpha^\prime$) and a vertical deviation angle $\angle$COC$_{\rm 1}$ ($\beta$$^\prime$).

As shown in Figure 4, there are five geometric relationships between the actual and estimated gaze point when the deviation angle is horizontal-right. Equation (2.1) can be inferred from the geometric relationship in Figure 4(a).

Then, we can get Eq (2.2) from Eq (2.1).

Equation (2.3) can be inferred from the geometric relationship in Figure 4(b).

Similarly, Eq (2.4) can be get from Eq (2.3).

The point A coincides with B, so $x_A = x_B$. Equation (2.2) can be re-written based on Eqs (2.4) and (2.5).

Equation (2.6) can be obtained based on the geometric relationship shown in Figure 4(c).

From Eq (2.6), we can get Eq (2.7).

The point A coincides with B$_1$, so $x_A = x_{B_1}$. We can get Eq (2.8).

Equation (2.9) can be obtained according to the geometric relationship illustrated in Figure 4(d).

Moreover, Eq (2.10) can be obtained according to the geometric relationship shown in Figure 4(e).

As shown in Figure 5, there are also five geometric relationships when the deviation angle is horizontal-left. The derivation process in Figure 5 is similar to Figure 4. It can be verified that all subfigures in Figure 5 are all satisfied for Eq (2.11).

It can be written as,

As shown above, the deviation angle is written as ${\alpha}^\prime$, if the ${\alpha}^\prime > 0$, the deviation angle is horizontal-right; otherwise, the angle is horizontal-left. It can be inferred as the following.

According to the above equation and $x_J = x_B$, it can be inferred as following.

Next, $\tan ({\alpha}^\prime)$ can be derived from Eq (2.13), and $x_J$ can be calculated from Eq (2.14) for gaze correction.

In the vertical section, there is a similar geometric relationship to the horizontal section. And the derivation process is similar. The deviation angle is written as $\beta^\prime$, if the $\beta^\prime > 0$, the deviation angle is vertical upward; otherwise, the angle is vertical downward. It can be inferred as the following.

2.2.   Gaze correction for myopia and strabismus

Based on the eyeball model of strabismus mentioned in Section 2.1, a sphere with larger diameter is adopted to model myopia in the hybrid eyeball model of myopia and strabismus. As shown in Figure 6, there are 7 geometric relationships between the estimated gaze point and the predefined point in case of myopia and horizontal-right strabismus. A, B, B$_1$ and B${_1}^\prime$ are collinear. DE is the moving distance of the pupil in the horizontal direction when the actual gaze moves from A to B in case of strabismus, and HI is the mapping of DE in the general eyeball model, and $DE = HI$. The moving distance of the pupil in the horizontal direction when the subject looks from A to B${_1}^\prime$ in the case of myopia and strabismus is HI.

As shown in Figure 6(a-c), $\triangle ODE$ is similar to $\triangle OB_1A$, and $\triangle OHI$ is similar to $\triangle O{B_1}^\prime A$, it can be inferred as the following.

As $DE = HI$, the radius OH of the reference eyeball model is written as r, and the radius OD of the actual eyeball model is written as R. Then, it can be inferred as Eq (2.18).

The coordinates of A, B, B$_1$ and B$_1^\prime$ are $(x_A, y_A)$, $(x_B, y_B)$, $(x_{B_1}, y_{B_1})$ and $(x_{{B_1}^\prime}, y_{{B_1}^\prime})$, it can be inferred as Eq (2.19).

It is known that Figure 6(d) satisfies Eq (2.19) when A, B and B$_1$ coincide.

As shown in Figure 6 (e–g), $\triangle ODE$ is similar to $\triangle OAB_1$, $\triangle OHI$ is similar to $\triangle OA{B_1}^\prime$, it can be inferred as the following.

Then, Eq (2.21) can be derived from Eq (2.20).

It can be known that all of geometric relationships in Figure 6 could satisfy Eq (2.19).

As shown in Figure 7, there are seven geometric relationships between the estimated gaze point and the predefined point in case of myopia with horizontal-left strabismus. The derivation process in Figure 7 is similar to Figure 6, it can be verified that all subfigures in Figure 7 satisfy Eq (2.19).

Similarly, The deviation angle is written as ${\alpha}^\prime$, if the ${\alpha}^\prime > 0$, the deviation angle is horizontal-right; otherwise, the angle is horizontal-left. It can be known that the varied conditions shown in Figures 6 and 7 could satisfy Eq (2.13).

Let $w = x_B-x_A$, $x_{B_1}$ can be deduced from Eq (2.13).

Then, introduce Eq (2.22) to Eq (2.21), we can obtain Eq (2.23).

When using the eye tracker, the coordinates of A, B and length OA can be derived in the step of calibration. B${_1}^\prime$ is the estimated gaze point given by the eye tracker, while $r/R$ and $\tan({\alpha}^\prime)$ is set as a constant for one eye.

The line-of-sight could meet the similar geometric relationship in the conditions shown in Figure 6 or Figure 7 after the eye tracker calibration.

Assume,

Then, introduce Eq (2.24) to Eq (2.23).

$M$ and $w$ are same positive and negative, it can be inferred as following.

As $w = x_B-x_A$ and $x_J = x_B$, from Eq (2.26), it can be inferred as the following.

Equation (2.23) is used for calculating the $r/R$ and $\tan({\alpha}^\prime)$, and Eq (2.27) is used for gaze correction.

On the vertical section, there is a similar geometric relationship to the horizontal section.

Assume,

The data correction derivation process is similar to the horizontal section, and it can be inferred as the following.

Equation (2.29) is used for calculating the $r/R$ and $\tan({\beta}^\prime)$, and Eq (2.30) is used for gaze correction.

In summary, there would be three variants, $r/R$, $\tan({\alpha}^\prime)$ and $\tan({\beta}^\prime)$, in hybrid eyeball model of myopia and strabismus for the gaze correction. They can be estimated based on Eq (2.23) and Eq (2.29) with at least 3 different predefined points in the step of calibration. Moreover, the proposed gaze correction method could be available for three-point calibration, while nine-point calibration is usually performed in the applications of eye-tracker.

It is obvious that the eyeball model of myopia is a degraded case for that of myopia and strabismus, when strabismus angles ($\alpha^\prime$ or $\beta^\prime$) equal to 0$^\circ$. Thus, Eq (2.27) and Eq (2.30) can also be used in the gaze correction for myopia.

3.   Computer simulations and results

The computer simulations are conducted to validate the proposed model-based gaze correction method. The parameters in the computer simulations are set according to the experience of implementation of the eye tracker. The screen size is set as 23.1-inch and the resolution is 1920 $\times$ 1080. The distance from eye to the screen of computer monitor, that is OA in Figure 2, is set as 65 cm. The point A is assumed in the center of the screen of computer monitor. The eyeball diameter of adults is about 23 mm normally, and the refractive power can be increased by about 3D for every 1mm increase in the eye axis ^[10]. In the computer simulatios, the range of $R/r$ is set in the range of 1.0 $\sim$ 1.2 for the condition of myopia based on the clinical experience. The range of strabismus angle ($\alpha^\prime$ or $\beta^\prime$) is about 0$^\circ$ $\sim$ 10$^\circ$ ^[11]. The noise model of gaze data is derived based on the calibration experimental data and adopted to simulate the external interference and eye physiological tremor. In the horizontal direction, the disturbance of gaze data is described as a Gaussian function with a mean value of 0 $^\circ$ and a standard deviation of 0.45$^\circ$. In the vertical direction, the disturbance of gaze is described as a Gaussian function with a mean value of 0$^\circ$ and a standard deviation of 0.49$^\circ$. The screen is divided into nine parts of the same size, and the center of each part as the predefined point. They are marked as 1, 2, 3...9 from top to bottom, left to right. Considering the similarity of the horizontal and vertical sections, we take the horizontal-right in strabismus as an example for simulation. The deviation of gaze data is the mean distance ${\triangle d}$ between the predefined point and the corresponding estimated gaze point in follows.

Figure 8 shows the example results of gaze corrections using the proposed method for myopia ($R/r$ = 1.1, $\alpha^\prime$ = 0$^\circ$), strabismus ($R/r$ = 1, $\alpha^\prime$ = 5$^\circ$ in horizontal-right) and myopia and strabismus ($R/r$ = 1.1, $\alpha^\prime$ = 5$^\circ$ in horizontal-right). It is clear that the deviations of estimated gaze points are significantly decreased after using the proposed gaze correction method. It can be found that the deviations of estimated gaze points are correlated to the positions of gaze points in the condition of myopia.That is, the larger deviations can be observed for the gaze points farther from the screen center. It can also be found that the data deviations are affected by several factors including the distortions of eyeball model caused by eye diseases and the positions of gaze points. Moreover, the deviation may not be linear when it varies with the univariate before correction. Moreover, a scenario is carried out to verify the infer where $R/r$ is set as 1.1, the real gaze point is set in a fix position as 672 pixels on the left of the screen center, and the strabismus angle ($\alpha^\prime$) in horizontal-right is changed from 0$^\circ$ to 10$^\circ$. As shown in Figure 9, the deviations of gaze points decreases first and then increases with the change of the strabismus angle before the correction; and the deviations of the corrected gaze points are less than 44.05 pixels. It can be seen in Figure 8(c), most deviations of the obtained gaze points tended to be within the range of 35.88 to 137.72 pixels (0.84$^\circ$ to 3.22$^\circ$) before gaze corrections, while deviations of the corrected gaze points were in the range of 22.26 to 29.20 pixels (0.52$^\circ$ to 0.69$^\circ$). The mean deviation is changed from 86.43 (SD = 38) pixels to 27.08 (SD = 3.28) pixels, which is reduced by 68.67% after the correction.

4.   Conclusions

In this paper, we have proposed a model-based method with geometric solutions for gaze correction, which is effective in reducing the deviation in eye-tracking data caused by the eye diseases, such as myopia and strabismus. The deviation of estimated gaze data are geometrically analyzed based on the individual eyeball model. By integrating the solutions for different conditions, a geometric solution is derived and used for gaze data correction. The experimental results have demonstrated that the proposed method is effective, and the mean deviation of the corrected gaze data is reduce by 68.67%. Moreover, the proposed method has the potential to correct the eye-tracking data and the modeling error. Although this algorithm proposed in this paper is very effective for the gaze correction, it also has some limitations. It adopts the sphere model proposed by Shao ^[12] which is different from the actual eyeball to simplify the calculation. The adaptive eyeball model will be discussed in the future work. Finally, it is concluded that the model-based method with geometric solutions has provided a simple approach to correct the eye-tracking data for various populations with eye diseases.

Acknowledgements

The authors would like to thank the physicians from the Department of Ophthalmology, West China Hospital and the Department of Ophthalmology, the First Affiliated Hospital of Xi'an Medical University for their professional suggestions and constructive discussions on the eyeball model. This paper is supported by Sichuan Science and Technology Program (No. 2019YFS140).

Conflict of interest

The authors declare there is no conflict of interest.