Determination of the location of the needle entry point based on an improved pruning algorithm

1.   Introduction

During the COVID-19 (corona virus disease 2019) pandemic, health care workers have been vulnerable to exposure due to direct contact with patients with infectious diseases. When dealing with such cases, medical staff need to wear protective clothing, goggles, etc., resulting in inconvenient movement and difficult veinpuncture operation. Furthermore, there is a shortage of protective clothing and other protective equipment, which cannot be reused. It is difficult to address these physiological problems of medical staff ^[1,2]. The intravenous injection of a dorsal hand vein is a routine part of patient care for medical staff. The success rate of intravenous insertion by clinicians is 40% in hard-to-reach patient groups ^[3]. An automatic hand vein injection robot can effectively replace nursing staff to complete daily nursing tasks and effectively reduce their burden and risk of infection.

After the outbreak of the pandemic, how to effectively reduce the risk of infection of medical staff has become one of the most important research topics in the industry. A number of research teams at home and abroad have invested in the application field of automatic robotic injection. A team led by Rutgers University has created a blood collecting robot and conducted the first human clinical trial of an automated blood collection and testing device ^[3]. The team of Qi et al. in Tongji University in Shanghai developed an automatic needle and blood collection robot to meet the clinical needs of the pandemic, which not only reduces cross-infection but also carries out large-scale needle and blood collection work ^[4].

This paper focuses on the realization of the core functions of a dorsal hand vein injection robot by adopting a modular design approach. The non-contact dorsal hand vein injection system was decomposed into five modules according to different functions: dorsal hand vein imaging, dorsal hand vein detection and segmentation, deciding on the needle insertion location, mechanical arm control, and injection feedback. Dorsal hand vein imaging and dorsal hand vein detection and segmentation are the basis and prerequisite for a successful decision on the insertion point location. Therefore, this paper focuses on dorsal hand vein imaging, dorsal hand vein detection and segmentation, and insertion point location decision modules.

In view of the above problems, the purpose of this paper is to develop a hand-back vein injection intelligent machine, which is mainly focused on the detection and segmentation of the hand back, and the function of the decision. This paper mainly makes the following contributions:

(1) For the problem of lack of hand vein and vascular information and depth points, a collection device was set up to establish information databases of different age groups. The database covers dorsal hand vein images of patients aged 20 to 79 years, and images of dorsal hand veins of patients of different ages can better test the robustness of the proposed algorithm.

(2) In view of the problem that the texture of the back of the hand cannot be segmented, the AT-U-Net semantic segmentation model is proposed by combining the improved U-Net and non-local attention mechanism and upgrading the U-Net network. By introducing non-local, this increases the connections between feature points on the map, expands the field of vision between veins and blood vessels by mapping neural network characteristics of the hand, increases the connections between long-distance veins and blood vessels, and facilitates the separation of long-distance veins and blood vessels. Through the U-Netdown structure, the information of feature layer can be deeply fused, and the enhanced feature extraction network has better universality.

(3) In order to fully extract the skeleton of dorsal palm vein image, an improved Pruning algorithm, Pt-Pruning, was proposed. The entry and exit points were calculated by multi-factor comprehensive evaluation. It mainly calculates the cross section size of vein vessels, length of dorsal hand vein vessels and bending value, and calculates the score of each insertion point by weighting, so as to evaluate and find the most effective insertion point of vein injection and later insertion point, laying a solid foundation for automatic injection of dorsal hand vein image.

2.   Related work

2.1.   Venous detection

At present, many auxiliary methods have been developed for venous imaging technology at home and abroad. Infrared imaging technology is widely used in the medical field because the difference of infrared light reflection and absorption characteristics between blood and surrounding tissues is much greater than that of visible spectrum ^{[5,6,7,8,9,10,11,12,13]}. Infrared light source is used to provide imaging conditions for intravenous injection. After projection, blood vessel images on human tissues can be clearly displayed. With the rapid development of infrared imaging technology, the related applications gradually increase. Infrared imaging has two advantages compared with other technologies. On the one hand, infrared light easily penetrates the skin, revealing detailed information under the epidermis, especially information about subcutaneous tissues. On the other hand, compared with the absorption and reflectance of the visible spectrum, infrared light has a more prominent effect, and currently medical infrared imaging systems are imaging through this characteristic. Researchers and research teams have developed a variety of intravenous AIDS using infrared imaging technology, combining image recognition operations in the medical field with artificial intelligence. For example, veinlite LEDX, a vein developer developed by TransLite, uses this optical imaging feature to display the body's shallow veins, but the device requires a healthcare worker to hold it in hand. Complicated operation procedures, unstable vascular display effect, high cost and many other factors are the reasons why the equipment has not been widely promoted and used.

The near-infrared vein projector is similar to that of the near-infrared imager. The near-infrared image information of the body surface is obtained through the infrared imaging technology, and then the processed algorithm is projected to the corresponding area, for example, the veins and blood vessels are displayed on the skin surface. This design greatly facilitates the progress of vein identification technology to a certain extent and facilitates the operation of venipuncture by medical workers. Such as vein viewer (Christie Medical) and vein detection eye wear (Evena Medical) developed by foreign teams ^[14], such equipment has clear imaging and very fast imaging speed. However, due to its high procurement cost, Therefore, it has not been widely promoted and applied in China.

Compared with near infrared imaging, ultrasonic vein imaging is also widely used. Due to the limitations of human hearing, the range of hearing perception is usually 20-20, 000 Hz. Other sounds beyond that range are inaudible to the human ear and are therefore called ultrasound. Like ordinary sound, ultrasonic waves can travel in a certain direction and can penetrate specific objects. When ultrasonic waves encounter obstacles, they will form echoes. Ultrasonic waves produce different echoes when they encounter different obstacles. The instrument collects echo related data information and displays it on the screen, based on which the internal structure of the object can be fully understood. The medical field usually uses ultrasound to assist doctors in the treatment of diseases. There are many different types of ultrasound used in medical diagnosis ^[15]. Among them, B-type ultrasound is widely used. It can be detected by B-mode ultrasound to obtain the profile of human organs. This method has low cost and no adverse reactions, so repeated examination can be carried out. For B-ultrasound image and related processing algorithm, color ultrasound system is widely used in hospitals.

Color doppler ultrasound is a method of imaging tissue in the body using sound waves. It generally consists of two parts: color ultrasonic phenomenon and two-dimensional ultrasonic imaging. Ultrasound has many advantages and has been used to guide venipuncture by combining with ultrasound ^[16]. However, only clinicians with high puncture techniques and skilled ultrasound devices can successfully guide the final process of venipuncture. The procedure requires two health care workers, one to attach the probe to the relevant part of the patient and the other to operate on the ultrasound information. In ultrasonic images, due to the irregular characteristics of metal materials, it is easy to scatter, and it is difficult to observe the specific location information in real time according to the information. In order to shorten the distance between the ultrasonic instrument with the patient, the researchers designed a sonic flashlight device ^[17], the device will display the image directly on the patient's skin, but monitors and other devices installed on the ultrasonic probe, the technology is relatively limited, in the figure to match, need to pass the real anatomy ratio shows that the organizational structure, This operation will result in the image displayed in the relevant part of the patient is too small and difficult to see, and it is usually impossible to judge the relative position and perform effective puncture. Sonic window (BK Ultrasound, Analogic, Inc), a handheld device constructed in C-mode Ultrasound imaging, mainly punctures the arm and displays a vertical view of the container on the user interface directly above ^[18]. With the device, images can be displayed to realistic anatomical proportions to help medical staff perform venipuncture operations. The device has major advantages in real-time performance. It can automatically adjust the scanning depth according to the depth of blood vessels, so as to achieve the effect that blood vessels can be displayed in the center of the image and achieve better visibility. However, the device can only effectively identify thick blood vessels, and cannot recognize thin blood vessels in the application. On the other hand, because the display screen is small and the magnification of the device is too small, it has certain limitations in practical application.

In the field of dorsal hand vein segmentation, many research teams and experts have built databases for biometrics and tested algorithms in the database. Algorithms during testing, different algorithms have different results. Zhao ^[19] et al. used the filter to smooth the image and protect the edge after the operation of the filter. After filtering, the image can avoid part of the fuzzy situation. Then the image is enhanced by Gaussian filtering. The image is further enhanced by histogram equalization. For the study of venous vessels, Zhang ^[20] studied the problem of venous enhancement, and discussed and verified the influence and effect of histogram correction, fuzzy algorithm and wavelet transform on images. Traditional local threshold segmentation methods include Niblack, iterative threshold segmentation, Ostu algorithm and so on. Miura Naoto proposed the maximum curvature algorithm successively ^[21].

At present, most teams are focused on the detection of elbow veins. Jie et al. proposed an algorithm for semi-automatic blood collection by extracting precious veins from elbow ^[11]. The algorithm uses homomorphic filtering, contrast limiting adaptive histogram equalization to remove noise and enhance image contrast, and then realizes elbow vein detection by adaptive threshold segmentation, hole filling and open and close operation. Due to the different distribution of hand elbow vein and hand dorsal vein, the number of blood vessels in hand elbow vein is relatively small, the cross-sectional area of blood vessels is large, and the direction of blood vessels is relatively single. The number of dorsal hand veins is relatively large, the cross-sectional area of blood vessels is small, and the direction of blood vessels is complex.

The algorithmic flow of the elbow vein semi-automatic blood collection approach is shown in Figure 1.

2.2.   Insertion point decision for dorsal hand veins

For intravenous injection into the needle point in the location decision making problems, Xun et al. put forward a kind of used for image of hand vein injection automatic detection and labeling algorithms ^[16], this algorithm in segmentation longest vein image information loss, segmentation effect is not obvious, and there is no consideration of factors such as the veins cross-sectional area of the needle point location decisions. The experimental database of dorsal hand vein in this algorithm contains 150 pictures of dorsal hand vein, a relatively small number, and the acquisition equipment is greatly affected by external interference light source and light source angle. There are problems of image information loss and inadequate segmentation in the segmentation of the longest vein, and the influence of the cross-sectional area of veins on the insertion point is not taken into account. Nowadays, deep learning extensive research greatly promoted the development of artificial intelligence and machine learning, deep learning in many important areas of social development has its place, made a number of technical tasks have a breakthrough, especially in view of the great application prospect in medical areas, thus allowing more into depth study of the research team ^{[22,23,24,25]}.

The flow of the algorithm for the automatic detection and labeling of dorsal hand vein images is shown in Figure 2.

3.   Materials and method

3.1.   AT-U-Net semantic segmentation network model

As a convolutional neural network, U-Net can enhance the data and achieve a better image segmentation effect through a few images ^[26]. Compared with traditional neural networks, U-Net brings better generalization ability while reducing complexity. Based on this, the medical image processing network develops it as the basic network. However, it is difficult to obtain the semantic features between long-distance veins and vessels with different shapes in the dorsal part of the hand ^{[27,28,29,30,31,32,33,34,35,36]}. In view of this problem, this paper proposes AT-U-Net. All the databases used are self-built dorsal hand vein databases. For details, see Section 4.1.1.

The AT-U-Net model consists of four parts: Figure 3 provides the model structure.

Feature extraction using the backbone feature extraction network gives a total of five characteristics of the layer: feat1, feat2, feat3, feat4, and feat5. Firstly, the 64-channel convolution of [3,3] is carried out twice to obtain a preliminary effective feature layer of [512,512,64], and then 2 $\times$ 2 maximum pooling is carried out to obtain a feature layer feat1 of [256,256,64]. Then, the convolution of 128 channels [3,3] is carried out twice to obtain a preliminary effective feature layer of [256,256,128], and then 2 $\times$ 2 maximum pooling is carried out to obtain a feature layer feat2 of [128,128]. Then conv3: convolved 256 channels three times [3,3] to obtain a preliminary effective feature layer of [128,128,256], and then 2 $\times$ 2 maximum pooling to obtain a feature layer feat3 of [64,64,256]. Then: a preliminary effective feature layer of [64,64,512] is obtained by three convolution of 512 channels [3,3], and then a feature layer feat4 of [32,32,512] is obtained by maximum pooling of 2 $\times$ 2. Finally, a preliminary effective feature layer feat5 of [32,32,512] is obtained through three convolutions of 512 channels [3,3]. Since deep features of network contain many redundant features, which affect the recognition of effective features, attention mechanism is introduced in feat5 of deep features of network.

After feature extraction, feat5 is used to introduce the directed attention mechanism to obtain more feature information between the long distances of dorsal hand veins.

The non-local attention mechanism is mainly used to capture the long-distance dependency relationship of dorsal hand veins, establish the relationship between pixels of veins on dorsal hand vein images, and find this relationship to better segment the long-distance dorsal hand veins ^[37].

In formula (1), $i$ and $j$ are the spatial positions, ${x_i}$ is the characteristic layer feat5 of the network output, ${y_i}$ is the residual structure, $f$ function is to calculate ${x_i}$ and all related ${x_j}$, and $f$ function follows non-local mean and double edge filtering. $g({x_j})$ is the eigenvalue corresponding to the corresponding position, $C(X)$ is the normalization parameter.

The non-local structure is shown in the figure. T is the number of video frames, H and W are spatial resolution, $\theta$, $\phi$ and $g$ are the number of convolutions. And for the matrix multiplication $\theta$ and $\phi$, the dimension after the operation, after the softmax operation, the matrix multiplication operation is performed with the third branch, and the dimension is restored to the same dimension as the input, and the operation of the formula (2) is performed.

In the formula, ${Z_i}$ is a data stream, ${y_i}$ represents the residual structure, ${W_z}$ is usually convolved with 1x1, which becomes the same dimension as the original feature graph ${x_i}$, and ${x_i}$ is the feature layer feat5.In the enhanced feature extraction network, the U-Netdown operation is introduced to replace the original operation of copy and crop, so that the enhanced feature extraction network has better universality. Figure 5 shows the network structure of the U-Netdown module.

As shown in Figure 5, U-Netdown has two inputs, corresponding to initial effective feature layer feat4 and feat5. The upper effective feature layer corresponds to Input1, and the lower effective feature layer corresponds to Input2. First, input1 is downsampling, and then Input1 and Input2 are stacked. After the operation is completed, two convolution operations are carried out to complete the construction of a U-Netdown module, that is, the feature fusion of two effective features is completed. Figure 6 shows the enhancement feature extraction network.

In Figure 6, block1 performs the upsampling operation, block2, block3, and block4 are the same, which are all upsampling operations after U-Netdown, and block5 is the U-Netdown operation. U-Netdown feature fusion is beneficial to improve the prediction performance of the prediction network.

In the prediction network, when the input image is rectangular, its shape needs to be changed to a square. If the resize operation is performed directly, the image can easily be distorted. In this case, the input image is called letterbox_image to ensure that it is not distorted. The letterbox_image operation gives the input image a gray bar that needs to be cut off before output. After this, we judge the type of each pixel, assign a specific color to each pixel, and convert the format of the segmented image to obtain the final prediction result.

The segmentation effects of the venous and vascular semantic segmentation network model are shown in Figure 7(a), (b). Figure 7(a) is the original picture of dorsal hand veins and vessels and Figure 7(b) is the dorsal hand veins and vessels segmentation map obtained by invoking the AT-U-Net semantic segmentation model.

3.2.   PT-Pruning: deciding on the dorsal hand vein insertion location

For dorsal hand vein injection, medical staff should tighten the skin at the lower end of the patient's vein with their left thumb and fix it, hold the syringe with their right hand, and insert the needle along the vein direction with an angle of 15°~20°, up to the skin. According to the analysis of puncture characteristics, a suitable vein for injection has three main conditions: 1) The vein has a certain length; 2) The degree of venous curvature is small; 3) The cross-sectional area of venous vessels is large. The three conditions can be matched to meet the requirements of the injection.

After the dorsal hand vein and blood vessel segmentation map is obtained, the veins and blood vessels are striped and labeled, and PT-pruning is used to make the decision on the insertion point. First, PT-pruning is used to remove burr branches from the refined vascular skeleton image to obtain the longest vein vascular skeleton main line. Then, the straightest area is extracted and its bending value is calculated by traversing the main artery of the vascular skeleton through a sliding window of fixed size. Next, the vascular segmentation map of the corresponding area is extracted and the cross-sectional area of the vessels is calculated. The barycentric coordinates of the veins in the selected area are taken as the insertion point. An entry point is determined for each blood vessel. Finally, the bending value of the corresponding area and the score of the vascular cross-sectional area are weighted to calculate the entry point of each blood vessel. The entry point with the highest score is selected and recorded as the optimal entry point of the dorsal hand vein, while the other entry points $pt0$ are recorded as effective entry points $pt1$, $pt2$, $pt3$.

Figures 8 and 9 show the specific flow chart of PT-Pruning.

The basic idea of PT-pruning is as follows: First, the nibbling method is used to determine the endpoints, solitary points, and encounter points in the skeleton, an undirected weighted graph data structure is used to store and refine the skeleton image, and then the longest path of the undirected weighted graph is calculated and determined as the skeleton main line. Here, nibbling is the elimination of a skeleton point in the skeleton whenever a skeleton point type is determined. The type of skeleton point can be judged by the cross number and striation number in the neighborhood of skeleton point 8. The vertex of the skeleton undirected weighted graph structure is the first endpoint, solitary point, and encounter point, and the weight of the edge is the length of the path between two vertices. The longest path can be determined by calculating the diameter of the undirected graph. It is divided into four stages: nibbling stage, calculation stage, reconstruction stage, and position decision ^[38].

(1) The nibbling stage.

The specific process of the nibbling stage is shown in Figure 10.

(2) Calculation stage.

After the nibbling phase is completed, a complete skeleton undirected weighted graph $G$ will be obtained, in which the vertex $G$ is the first end point, solitary point, and encounter point of the skeleton, and the weight of the edge is the length of the path between the two vertices.

Since the backbone skeleton is generally the longest vessel, the computation of the skeleton can be converted to searching for the longest path in the undirected weighted graph $G$. The distance between the two vertices farthest apart in a dorsal hand vein diagram becomes the diameter of the graph, so this paper obtains the longest path by calculating the diameter of the undirected graph. The algorithm is described as follows: Select any vertex $s$ in the graph and find the vertex $x$ farthest from it; find the farthest vertex $y$ of $x$, the distance between vertex $x$ and vertex $s$ is the diameter of the graph.

At the end of the calculation phase, we obtain the vertices contained in the longest path ${e_{longest}}$ of the undirected weighted graph which are also contained in the backbone of the vascular skeleton.

(3) Reconstruction stage.

In the nibbling stage, all non-zero pixels in the image will be nibbled away, so we get a blank image after nibbling. In the reconstruction phase, we start from the first vertex $v$ of ${e_{longest}}$, visit the edge ${e_{longest}}$ (adjacency list node) of the vertex that is adjacent to $v$ and belongs to $e$, read the pixel set $pts$ contained in this path from $e$, and set all pixel points $pts$ H to 255. This process continues until the last vertex of ${e_{longest}}$ is reached. The vascular pruning effect for the dorsal hand vein is shown in Figure 11(a), (b).

Figure 11(a) is a vessel in the dorsal vein of the hand, and Figure 11b is the arterial vein skeleton after PT-Pruning.

After the arterial vein skeleton is obtained, the number of pixels contained in it is denoted as the length of venous vessels, $X$. Then, a fixed size sliding window is used to traverse the skeleton trunk line, and the bending value of blood vessels in the sliding window is calculated. The calculation method is as follows: the main vein skeleton in the sliding window area is fitted into a straight line, and formula (3) is used to calculate the sum of distances between points on the skeleton line and the straight line, which is defined as the bending value of vessels in this section, denoted as $Y$.

In the formula $Y$, $A$, $B$, and $C$ are the value of vascular curvature; ${x_i}$, ${y_i}$ is the coordinate of point $i$; $n$ is the number of points on the skeleton line in the sliding window.

The original image of veins in the straightest region where the sliding window is located can be obtained by sectioning. The number of pixels contained by veins in this region is denoted as the cross-sectional area of veins and is denoted as $Z$. The cross-sectional area of blood vessels is determined by the number of pixels contained. The more pixels contained, the larger the cross-sectional area of blood vessels in this part. The barycenter of veins in the image of this region is selected as the entry point, and one entry point is selected for each vessel. Finally, a decision needs to be reached, and a normalized method is used to calculate the bending value and cross-sectional area of the straightest area of each vein and vessel according to formula (4), which is based on a weighted decision algorithm of the vein and vessel entry point, and the scores are sorted. The insertion point with the highest score is denoted as the optimal insertion point ($pt0$), and the other insertion points are denoted as the effective insertion points: $pt1$, $pt2$, $pt3$, ..., $ptn$ (n is the number of veins).

In formula (4), $pt$ is the score of the insertion point; $A$, $B$ and $C$ are the coefficient of the linear equation; ${X_i}$ is the length of the arterial skeleton after pruning algorithm processing; ${Y_i}$ is the bending value of the arterial skeleton in the selected specific region; ${Z_i}$ is the cross-sectional area of the arterial veins in the selected specific region; $n$ is the total number of veins.

The effective flow of the PT-Pruning algorithm is shown in Figure 12.

Operation 1 in Figure 12 splits different vessels in the venous vessel segmentation diagram to obtain independent venous vessels. Operation 2 removes burr branches of vein vessels by PT-Pruning to obtain the longest skeleton trunk of the vein vessels. Operation 3 uses the sliding window of fixed size to traverse the vascular skeleton main line, calculates the distance between the skeleton main line and the fitting straight line in the sliding window, obtains its bending value, and selects the region with the minimum bending value. The bending value in the left picture is 20.5277, and the bending value in the right picture is 20.5056. Operation 4 is the blood vessel image corresponding to the region with the smallest bending value, and the cross-sectional area of the corresponding blood vessel is calculated. The cross-sectional area of the blood vessel in the left picture is 1877.5, and the cross-sectional area of the blood vessel in the right picture is 2356.0. Operation 5 takes the center of gravity of blood vessels in the selected area as the insertion point and displays its position coordinates in the original image. The insertion point position of the left picture is (499, 341), and the insertion point position of the right picture is (597, 210). After obtaining the entry points of each vessel, operation 6 sorts the entry points by calculating the bending value and cross-sectional area of the entry points of each vessel, and obtains the optimal entry points with coordinates of $pt0$ (499, 341) and $pt1$ (597, 210). The pruning algorithm consumes 10-20 ms.

4.   Experimental verification

4.1.   Dorsal hand vein database

4.1.1.   Setting up the dorsal hand vein collection device

The core technology of handlet vein injection is venous vessel testing and entry point position decision, and the database is the key to overcome the core technology. Since there are few disclosed data sets, the data set is constructed based on the multi-optical spectral handlet venous acquisition system that is independently developed.

Hand venous venous imaging is different due to illumination conditions, and its imaging effect is also different, and it is ideal for the use of ordinary photographic vein vascular imaging. There is a large number of hemoglobin in venous blood vessels, and hemoglobin is more likely to absorb 700-1000 nm under near-infrared light, and the imaging characteristics is more obvious than the subcutaneous tissue below the blood vessel. Under this imaging principle, the experiment was used as an image acquisition device using 730, 850 and 940 nm. The specific device is shown in Figure 13.

The dorsal hand vein imaging platform consists of four parts. The first part is a CCD black and white camera which is used to obtain the dorsal hand vein image. The second part is the band pass optical filter, which can reduce the influence of other wavelengths of the other wavelengths of light, and reduce the effect of other wavelengths of the light intensity, making the characteristics of the blood tube more clear and increasing the accuracy of the camera's photography. The third part is a multi-spectral array and homogeneous plate. The multi-spectral array is uniformly arranged around the CCD camera to provide sufficient light source for the imaging of dorsal hand veins. In order to prevent the light source from concentrating too much and forming a light spot, a homogeneous plate is placed in front of the multi-spectral array. The fourth part is the hand-held infrared light display instrument. When collecting venous images, a person puts their palm on the display instrument. Infrared light can pass through the palm to improve the vein and vascular imaging performance and address the problem of vascular hiding and imaging not obvious in patients.

Experimental objects are widely distributed, and the agenda is distributed in 204-bit genders, the age of 20 and 79 years old, collecting left hand and right hand back vein images, handlet vein image data set contains 2456 pictures. The data set was divided according to the ratio of 9:1. Among them, the training set contained 2210 images and the test set contained 246 images. A total of 2456 images were tagged, and 2456 experiments were conducted for each method.

The image is shown in Figure 14.

4.1.2.   Establishment of dorsal hand vein database

The original images in the database are divided into two parts: vein and blood vessel, and background. The label of the former is achieved by marking the vein and blood vessel on the back of the hand pixel by pixel. The original database images and label images are shown in Figure 15.

4.1.3.   Establishment of dorsal hand vein insertion point database

The original pictures in the database were divided into two parts: optimal injection point and effective injection point. Each blood vessel was marked with an injection point, and the optimal injection area and effective injection area in the dorsal vein of the hand were marked. The optimal injection area was marked as, and the effective injection area was marked as $pt1$, $pt2$, $pt3$, etc. The original database images and label images are shown in Figure 16. The data set was divided according to the ratio of 9:1. Among them, the training set contained 2210 images and the test set contained 246 images. A total of 2456 images were labeled, and a total of three specialists labeled venous entry points. 2456 experiments were performed for each method. The 5 groups of images in the figure are collected by experts according to the dorsal hand vein and incorporated into the medical knowledge of needle point puncture. The labeled dorsal hand vein enters the needle point. The first group was marked with one entry point, which was the optimal entry point; the second group was marked with two entry points, one optimal entry point and one effective entry point; the third group was marked with three entry points, one optimal entry point and two effective entry points. The fourth group was labeled with two entry points, one optimal entry point and one effective entry point. The fifth group marked one insertion point as the optimal insertion point.

4.2.   Evaluation indicators

There are three evaluation indicators of hand back venous blood vessels, respectively: MIOU, F1-score, MPA and the calculation formula is:

In the formula (5), TP and TN are classified correct blood vessels, background pixels. FP and FN are categorical erroneous blood vessels, background pixels; F1-score in formula (6) is similarity to the handlet vein database, the ${P_i}$ in the formula (7) is the correct rate of each category, $num\_classes$ is the number of categories.

There are two evaluation indexes for needle entry point position decision, and the calculation formula is:

In formula (8), ${P_1}$ is the detection accuracy of effective needle entry points, TP is the number of valid needle entry points correctly classified, and FP is the number of effective needle entry points incorrectly classified. In formula (9), ${P_2}$ is the detection accuracy of optimal needle entry points, TP is the number of correctly classified optimal needle entry points, and FP is the number of incorrectly classified optimal needle entry points.

4.3.   Dorsal hand vein segmentation experiment

The CPU model of the experimental platform is Intel(R) Core(TM) I5-4590, the memory size is 8GB, and the GPU model is GeForce GTX1050.

The segmentation experiment for dorsal hand veins is here divided into two parts. The first part is the experiment using the AT-U-Net semantic segmentation model, and the second part is the comparative experiment which consists of experimentation involving the traditional algorithm and other semantic segmentation models.

AT-U-Net is used to detect and segment venous vessels in the hand dorsal vein database, as shown in Figure 17.

As can be seen from Figure 16, AT-U-Net can segment the dorsal vein vessels of the hand for a long distance, providing a basis for the position decision of the needle entry point in the next stage.

The second part of the experimentation is a comparative experiment. Through this, it is found that the traditional algorithm has higher requirements on hand posture, and the experimental results are shown in Figures 18(a)-(f). Figure 18(a) shows the original image of the back of the hand. Figure 18(b) is the result after homomorphic filtering of the image. Figure 18(c) is the result after CLAHE processing. Figure 18d is the result after applying the adaptive threshold segmentation. Figure 18e is the image after dealing with the morphology. Figure 18f is the resulting image after dealing with the closed operation. It can be seen in the figure that the left side of the blood vessels cannot complete extraction. Moreover, the partial distortion of vascular endpoints is serious, with much noise, and the effective area is too short, which is not conducive to the later insertion point detection.

Compared with the experimental network model, the PSPNet network is a kind of small and sufficient depth feature, but its shallow layer characteristics are not enough, and the detail segmentation effect is poor. But the effect of the long distance semantic feature is unknown. SegNet can maintain the integrity of the high frequency information, but it also ignores the information between adjacent veins when it is treated with a low rate of hand back vein. The inner bending of RefineNet means that it has lost some of the details of the back of its hands. PI-Unet is an improvement network that is divided into heterogeneous and complex images that can be obtained under small amounts of data, which can be applied to low-performance marginal computing devices, but the blood tube is longer than the length of the tube, which is not very good for extracting the long distance hand back blood tube. In this paper, the deficiency of the above model, and the blood tube of the remote hand vein, can be extracted, and the problem of the uneven distribution of the long distance blood tube is solved ^{[39,40,41,42,43,44]}. In order to evaluate the performance of each network model, this paper analyzes the evaluation indexes, and the specific records are shown in Table 1. Each algorithm in this table has conducted 2456 experiments, and the results shown in this table are the average of all experiments.

As can be seen from Table 1, using the same dorsal hand vein database, the same initial learning rate, the same number of training rounds, and the same batch size, the segmentation performance of each semantic segmentation model is different. The AT-U-Net semantic segmentation model achieved a high score in all three indexes, which has advantages in the segmentation of long-distance dorsal hand veins.

The model-specific segmentation results are shown in Figure 19.

As can be seen from the predictions in the figure, PSPNet has the lowest performance in vein and vessel segmentation, with inaccurate vascular structure segmentation and a large deviation in the segmented vessel cross-sectional area compared with real vessels. The width of U-Net vein segmentation is similar to that of real blood vessels, but discontinuous breakpoints will appear when it is segmented into long veins. SegNet has poor segmentation performance for adjacent dorsal hand veins, RefineNet has incomplete segmentation for the details of the dorsal hand veins, and PI-Unet has complete segmentation for details of veins but incomplete segmentation for long-distance vein detection. Compared with the segmentation performance of the above models, the segmentation performance of the AT-U-Net network model is a significant improvement. The cross-sectional area of the veins segmented by the AT-U-Net network model is similar to that of real vessels, and it can handle the details of the segmentation between adjacent vessels, which has obvious advantages in the segmentation performance of long-distance dorsal hand veins.

4.4.   Decision of insertion point position

The experimentation concerning the decision regarding the needle insertion position is divided into two parts. The first part considers PT-Pruning and the second part is a comparison with the automatic labeling and recognition of dorsal hand vein images proposed by Zhang et al.

After obtaining the segmentation of dorsal hand veins, PT-pruning is called to make the decision regarding the needle insertion position, as shown in Figure 20.

Figure 20 shows that PT-Pruning can accurately locate the insertion points of dorsal hand veins, providing a solid foundation for subsequent automatic injection.

In order to demonstrate the effectiveness of the algorithm objectively and fairly, 2546 samples are verified one by one to identify the effective region and the optimal injection region. By comparing with the optimal region and effective region in the database, the obtained insertion point location is verified. In detecting whether or not this is in the effective area, the algorithm calculates the effective area (including the optimal injection area into the needle point) and this is compared with that in the database. The verified the results are shown in Table 2. In order to better detect the accuracy of the algorithm, the pin insertion position obtained by the algorithm is compared with the database, if the optimal injection point is in the optimal injection region in the database, it is judged as a correct identification of the optimal injection point; if it is not in the optimal injection region, it is regarded as a wrong identification of the optimal injection point. The verification results are shown in Table 3.

It can be seen from Table 2 that the accuracy of identifying the entry points in the effective injection area is 96.73%. It can be seen from Table 3 that the accuracy of identifying the entry points in the optimal injection area is 96.5%, indicating that PT-pruning shows good detection performance.

In order to make a more intuitive analysis of the performance regarding needle insertion position in each experiment, this paper made statistics of the experimental effect diagrams, as shown in Figure 21.

In Figure 21, the first column is the optimal injection area and effective injection area of the needle insertion point labeled by the database; the second column is the experimental result of automatic labeling and recognition of the dorsal hand vein images; the third column is the experimental result obtained by PT-pruning. Through observation, PT-pruning has the following two advantages. The first advantage is that PT-pruning can accurately locate the specific coordinates of the entry and exit needle points. The second advantage of accurate recognition area, the back of hand vein image automatic labeling and identification of block matching blood vessels to the setting of a fixed area, not the global detection on veins, PT-pruning by traversal vein injection region extraction, can carry on the overall detection to the vein and the vein cross-sectional area as well as the bending value comprehensive decision. The needle point area obtained by PT-pruning is more suitable for intravenous injection, and has strong effectiveness and practical application. It provides a solid theoretical basis for the automatic injection of dorsal hand veins.

5.   Discussion

It is of great significance to develop and popularize the intelligent robot for hand dorsal vein injection in application scenarios. The current stage is for infectious ward research and development. In the following stage, other scenarios will be gradually developed for its function of reducing excessive medical personnel, such as: in remote areas and wild areas with relatively insufficient medical resources, special places such as the space station can be used as emergency first aid tools to carry out some basic treatment urgently and quickly, providing sufficient time for follow-up professional treatment and preventing life from being in a critical state due to the rapid decline of body function.

In terms of functional development of the intelligent robot for dorsal hand vein injection, the current stage is mainly about basic function research and development, and the dorsal hand vein database can be improved in the next stage. Due to the current epidemic period, there are sporadic outbreaks in various regions. According to the current epidemic prevention policy, and the data collection process of infants and young children is cumbersome, so the current database of dorsal hand vein does not involve the information of the blood vessels of infants and young children, which makes it impossible to evaluate the effect of intelligent injection of dorsal hand vein for infants and young children. In the next stage, the vein injection area of the back of the hand can be gradually extended to the elbow vein and other locations to further improve the robustness of the intravenous robot. According to the distribution of functional modules of the robot, the algorithm is gradually transplanted to the robot arm, and the automatic intravenous injection of the back of the hand without manual contact is finally realized. The organic combination of internet plus and medical treatment is truly realized, so that the intelligent technology can better serve medical treatment.

6.   Conclusions

During the epidemic, the amount of hand induced by the vein is increased, and this paper conducts research on its core issues, opposing the venous blood vessels and the position decision problem of the needle point. For hand-venous venous vessel segmentation, this paper proposes an AT-U-Net network model that extracts long-distance handlet venous vessel characteristics and divides them. For deciding on the needle point location, in this paper, on the basis of the segmented image, a PT-pruning decision method was proposed. The pruning algorithm was used to extract the skeleton vein trunk, then the length of the vein skeleton trunk and skeleton were calculated through a sliding window traversal. The straightest area was extracted and its bending degree was calculated. The blood vessel segmentation map of the corresponding area was further extracted and the cross-sectional area of the blood vessels was calculated. The barycentric coordinates of the veins in the selected area were used as the insertion point. Finally, the score of the length, curvature, and cross-sectional area of each blood vessel was weighted, and the insertion point with the highest score was selected as the optimal insertion point; the other insertion points were regarded as effective insertion points. However, the algorithm proposed in this paper still has some shortcomings.

Acknowledgments

The research was supported by the Shandong Provincial Natural Science Foundation project, the research on key technologies for network public opinion response and governance of major public health emergencies, project number: ZR2021MF064. Five anonymous reviewers have carefully read this paper and have provided to us numerous constructive suggestions. As a result, the overall quality of the paper has been noticeably enhanced, to which we feel much indebted and are grateful.

Conflict of interest

The authors declare there is no conflict of interest.