Device performances and instabilities of the engineered active layer with different film thickness and composition ratios in amorphous InGaZnO thin film transistors

1.   Introduction

As a minimally invasive interventional surgery, percutaneous puncture has the advantages of minimal trauma, fast recovery and strong targeting ^[1], and it is widely used in human tissue biopsies and drug-directed treatment. Among the various procedures, lumbar puncture is often used for the diagnosis of various central nervous system inflammatory diseases, vascular diseases, spinal cord diseases, intracranial-space-occupying diseases, unknown nervous system diseases and spinal angiography. In addition, lumbar puncture is also a common clinical diagnosis and treatment method for drug injection treatment and decompression due to the high cerebrospinal fluid pressure caused by central nervous system diseases ^[2,3].

In traditional lumbar puncture surgery, doctors use medical imaging equipment and their clinical experience to determine the location of the needle puncture point on the skin. Medical imaging mainly includes computed tomography, magnetic resonance imaging and ultrasound imaging ^[4]. Compared with computed tomography and magnetic resonance imaging, ultrasound imaging has the advantages of being low cost, no radiation and real-time image generation ^[5]. Therefore, it has become the best choice for doctors in lumbar puncture surgery. However, the time and results required to complete a scan are dependent exclusively on the skills and experience of the doctor. In addition, ultrasound scanning requires a complex posture of the hands of doctors to maintain a certain contact force; hence, long-term work may cause complications such as arthritis ^[6,7]. The use of electric power to operate a mechanical arm can reduce the burden on doctors.

During an operation, the robotic arm puncture system has the advantages of high precision, good stability and simple and efficient operation ^[8]. Aiming at the clinical needs of lumbar puncture, the present study involved the design of a robotic-assisted ultrasound scanning system for lumbar puncture. The computer-controlled robotic arm clamps the ultrasound probe and automatically scans the patient's lumbar spine.

2.   Materials and method

2.1.   System software design

The proposed system is shown in Figure 1, and it mainly included DP-50 portable ultrasound diagnostic equipment (Shenzhen Mindray Biomedical Electronics Company) for acquiring the two-dimensional ultrasound images, a six-degree-of-freedom robotic arm (UR3, Universal Robots A/S, Odense, Denmark) for controlling the movement of the ultrasound probe, a main control computer with an Intel Core i7-7700 processor to control the robot arm and process the image, a video capture card (Epiphan DVI2USB3.0) for acquiring two-dimensional ultrasound images and transmitting them to a computer and a GRIPKIT-E gripper (Weiss Robotic Company) for holding the ultrasonic probes.

Figure 1.  Lumbar spine and visualization system hardware components.

DownLoad: Full-Size Img PowerPoint

The system was divided into three main parts: control, acquisition and visual processing. Among them, the control part mainly involved the main control computer (including software), the robot arm control box and the structure of the robot arm; the acquisition part mainly involved the portable ultrasonic scanning system and the video capture card; the visualization processing was mainly realized through the software on the main control computer in which the code was embedded. In the operation of the system, the operator controlled the control program interface through the peripheral equipment of the main control computer and monitored the running state of the system in real time through the display, and the robot arm and the control box communicated with the computer through the Transmission Control Protocol/Internet Protocol. When the operator determined the starting point and end point of the scan, the control program automatically controlled the robot arm holding the ultrasound probe to scan the lumbar vertebrae and collect two-dimensional ultrasound images. The collected ultrasound images were transmitted to the main control through the data acquisition card. The computer then processed and saved the images. After the scan was completed, the visualization toolkit (VTK) 3D image reconstruction program was employed to perform the 3D reconstruction of the scanned images. The reconstruction result can provide a reference and guidance, thereby enabling the doctor to determine the puncture point.

2.2.   System hardware design

The control software was developed by using the Microsoft C# programming language, which mainly included five modules. The user interface module (LumbarPunctureSystem) was the main module of the system; it had the main following functions: parameter setting, operation control, process control, image display and system status display. The surgery module (SurgeryModule) was used to create new cases, hardware (robot arm, ultrasound) start and control and system settings. The robot control module (RobotModule) realized the motion control of the robot arm so that the robot arm could independently move along the axes of the base coordinate system and the tool coordinate system. The ultrasound imaging module (ImagingModule) was used to collect the ultrasound images and record information, such as the position of the robot arm and the size of the torque during the acquisition process, and to perform 3D reconstruction after the acquisition. The button control module (EnhancedGlassButton) was used to create button controls for the interface.

Initially, the control software was launched; the program automatically connected the robot arm and the acquisition card. After the connection was successfully established, the new case button in the operation control area was activated. The operator created a case and filled in relevant information by clicking the new case button. Subsequently, after the successful creation, the tool button was activated.

After the operator clicked the tool button, the program controlled the robot arm to grab the ultrasonic probe placed at a fixed position, activated the button to set the scanning start point and prompted the robot arm motion control panel to appear on screen. The operator adjusted the robot arm through the control panel by moving it to the starting point of the model scan and pressed the start button to activate the scanning end button. Similarly, the probe could be moved to the end point, after which the end point button could be pressed. Subsequently, the program launched the scan control interface, which displayed the corresponding information and supported the modification of some settings. Finally, when the operator pressed the scan button, the program automatically controlled the robot arm to move along a specific path from the scan start point to the scan end point and performed real-time image storage. During the scanning process, every time the robot arm moved 0.5 mm, the program acquired an ultrasound image, as shown in Figure 2. During this process, the robotic arm was set to run in force mode to ensure that the probe was in effective contact with the measured object. After artificial planning, the robotic arm moved in the lumbar spine direction and it automatically adjusted the position in the vertical direction, following force-limited parameters in grip, thereby acquiring a good ultrasound image. It is worth noting that, before the scan started, information such as the pressure of the robot arm and the interval of image acquisition was set on the scan control interface. In addition, once the new case was created, the program automatically activated the stop button in the operation control area. At any time during the scan, the program could be terminated by pressing this button to prevent accidents and ensure the safety of the scan.

Figure 2.  Two-dimensional ultrasound image acquisition program following robot arm scanning (the active green strip is the force status bar of probe).

DownLoad: Full-Size Img PowerPoint

2.3.   Lumbar spine model experiment

The lumbar spine model used in this study was the Enovo Medical Model (Shanghai, China) shown in Figure 3. Ultrasound has a good propagation speed in hydrogel material ^[9], and it produces a good coupling between the model and the probe. As a result, the experimental model for scanning was made in an acrylic material container (made in our laboratory), the spine model was put in it and the mixture of hydrogel and pure water was poured for lumbar region modeling. The liquid surface was 1–2 cm above the model. The ultrasonic probe model included a 75L38EA linear array, and the ultrasonic frequency was 8.5 MHz, the scan width was 3.8 cm and the depth was 4.6 cm. According to clinical experience, lumbar puncture is usually performed at the 3rd to 4th lumbar vertebral space, and in children under 4 years old, it needs to be punctured at the 4th to 5th spinous process space of the lumbar vertebrae. Therefore, the scanning range of this experiment was L3–L5. In the experiment, the main control program controlled the robot arm to clamp onto the ultrasonic probe to determine the scanning start and end points. The acquisition interval (0.5 mm) and force mode value (2N) were set to perform the two-dimensional ultrasound image acquisition. Then, the obtained image was processed for the 3D reconstruction and visualization.

Figure 3.  Lumbar spine model: A. Model from Enovo Medical Model Company, B. Reprocessing lumbar spine model for ultrasonic probe scanning.

DownLoad: Full-Size Img PowerPoint

2.4.   Three-dimensional reconstruction

After successfully acquiring the two-dimensional ultrasound tomographic image, the 3D reconstruction button in the main control program was activated. The operator pressed this button to perform the VTK-based 3D ultrasound image reconstruction and display, which can provide a reference and serve as guidance for determining the location of the puncture point and surgical path planning.

The VTK is an open-source visualization development tool developed by the Kitware Company, and it is widely used in the fields of visualization and 3D graphics, as well as other fields. The 3D visualization process generally involves medical image data reading, data preprocessing, visual mapping, reconstruction and interactive operation. Image data acquisition is a two-dimensional tomographic medical image sequence obtained by scanning real objects by using an imaging device. The format of the read image included the DICOM format and BMP format. Data preprocessing was carried out because the quality of the image may be reduced to a certain extent during the image output, transmission and conversion, and due to the introduction of noise. Hence, the image needed to be preprocessed by filtering. Visual mapping converted the processed raw data into geometric pixels and attributes for drawing, including the display shape, color and other attribute settings. Common methods for reconstruction include surface and volume rendering. Surface rendering describes the 3D structure of organs or tissues by fitting the surface information of the structure by data splicing, and the key technology sets a threshold to extract the isosurface according to the needs ^[10]. Volume rendering directly synthesizes the 3D images by resampling the volume data based on the principle of vision. This method treats each voxel as a particle that can receive or emit light, and it configures the light intensity and opacity, integrates along the line of sight and finally forms a projected image. Because surface rendering may lead to the reduction of detailed information, resulting in poor authenticity, this study adopted the use of the volume rendering method for 3D reconstruction. The interactive operation was performed on the basis of the appearance of the 3D reconstruction results. By clicking the mouse, the reconstruction body could be rotated to any angle.

The main methods of volume rendering are ray casting, maximum intensity projection and texture mapping ^[11]. Since the image quality obtained by the ray-casting algorithm is high, it is most widely used. The flowchart of the ray-casting algorithm is shown in Figure 4. The specific steps of the ray-casting method in the VTK were as follows. First, the bitmap (BMP) picture was read through the vtkBMPReader class. Subsequently, the vtkPiecewiseFunction class was used to determine the opacity of each voxel through the addition of numerical points, and the vtkColorTransferFunction class was employed to implement the color transfer function to add color values or a gray degree value. Then, by selecting different vtkVolumeMapper classes in the drawing pipeline, different volume rendering algorithms were implemented, among which the ray-casting algorithm used the vtkVolumeRayCastMapper class. Finally, the vtkVolumeActor class was used to specify the scene lighting, perspective, focus and other information, and the vtkRender class was used to render the entities in the drawn scene. The VTK also provides the widget base class. In this experiment, the vtkImagePlaneWidget class was used to create cutting plane objects, and the cross section, coronal plane and sagittal plane were successfully displayed.

Figure 4.  Flowchart of ray-casting algorithm during the three D reconstruction processing.

DownLoad: Full-Size Img PowerPoint

In this study, the two-dimensional ultrasound images collected by the control program were saved as an 8-bit BMP grayscale image. Subsequently, the reconstruction program was employed to visualize the 3D reconstruction, and the images were processed by using the median filtering method. According to the two-dimensional images collected every 0.5 mm, the ultrasound acquisition depth and probe width were set corresponding to the pixel points of the BMP image in order to calculate the difference between the data points. This was done to ensure that the image scale and size were consistent with the real object. Interactive operation was also set in the program. When the reconstruction result was obtained, the operator could place the mouse cursor in this area and hold down the left mouse button to move it arbitrarily. The operator could then rotate the 3D reconstruction body to any angle for full observation. The zoom function was also realized by rolling the mouse wheel.

3.   Results and discussion

A total of 136 two-dimensional ultrasound images were collected in the lumbar spine model experiment, and the dimensions (in pixels) were 393 × 473. After preprocessing by enhancing the contrast of and denoising the original ultrasound images, the ray-casting algorithm was used to perform the 3D reconstruction of the lumbar spine model. The reconstruction results are shown in Figure 5. Figure 5B is the 3D reconstruction result of the volume rendering ray-casting method, and Figure 5A, C, D are the cross section, sagittal plane and coronal plane after reconstruction, respectively.

Figure 5.  Three D reconstruction images of lumbar spine model after robot arm scanning and computer processing (L3, L4 and L5 are the lumbar spinous processes).

DownLoad: Full-Size Img PowerPoint

Repeated experiments with different entry points were performed and recorded, following the parameters above; the results are shown in Figure 6. The 3D images and corresponding sagittal plane slice images were labeled to show the reproducibility for this model.

Figure 6.  Three D images and corresponding sagittal plane slice images acquired for different entry points; 1a–5a are the three D images and 1b–5b are the corresponding sagittal plane slice images; 1c–5c are the location diagrams of the entry points (red points) and scanning direction (green arrows).

DownLoad: Full-Size Img PowerPoint

Due to the propagation characteristics of the ultrasound along a straight line, the information of the spinous processes perpendicular to the ultrasound linear array was lost. Thus, the reconstructed structure was defective; however, the location of the spinous process gap could be determined by using the sagittal and coronal images, which can be used as a follow-up reference basis for the puncture surgery. The repeated experiment data with different entry points (Figure 6) shows the anatomical structural changes in grey value and sharpness of boundary, which might have been caused by entry point changes, along with shape changing on the back during the scanning of the model.

In recent years, with the gradual maturity of the robotic technology, puncture surgery systems assisted by a robotic arm have developed rapidly. Nelson et al. ^[12] developed a set of ultrasound image-guided breast biopsy systems for the detection of early breast cancer; it was shown to have good repeatability and less than 0.5% variation. Podder et al. ^[13] designed a multichannel robot system based on the ultrasound guidance, and it comprised multiple particle implantation needles, especially for the implantation of radioactive particles in the prostate ^[14]. An experiment revealed a conveyor accuracy within 0.2 mm. Kojcev et al. ^[15] used both arms to design a set of ultrasound-guided auxiliary puncture systems; it implemented the closed-loop control of the planning-imaging-needle insertion to adjust and plan the puncture path in real time. Xu et al. ^[16] developed a minimally invasive robotic system for the ultrasound-guided microwave ablation of the liver. This system could perform 3D reconstruction of the liver tumor before surgery and plan the puncture path; it assisted the doctors in sending the puncture needle to the target point during an operation. The above-mentioned previous studies mainly involved ultrasound guidance and included location planning of the body or probe through the use of computed tomography, optical assisted technology and other imaging methods, instead of the robotic arm automatic guidance with force feedback. Our study provides an easier way to employ ultrasound scanning, find the precise location and simplify the complicated operation. In this study, the scanned ultrasound images were processed and reconstructed in all three dimensions. After experimental verification, the system was shown to be able to collect two-dimensional ultrasound images and reconstruct 3D lumbar spine images, thereby providing the doctors with stereoscopic visualization guidance, laying the foundation for subsequent robot-assisted puncture surgeries.

Regarding the usage of this methodology in actual clinical practice, more studies need to be done in the future, such as the use of an automatic drawing point and trajectory as a reference for the doctor, the addition of the modules for different types of conditions, like leukemia cerebrospinal fluid examination, scoliosis and larger body habitus and the redesign of different algorithms for robotic arm motion and puncture planning.

4.   Conclusions

The present study was purposed to design a 3D visualization-enabled robotic-assisted ultrasound automatic scanning system for the lumbar spine. The control program for the robotic arm was developed using the C# language, and a 3D reconstruction program for the ultrasound images was developed based on the VTK. The system controlled the robotic arm to automatically scan and collect the ultrasound images of the lumbar spine model through the program; it subsequently performed the 3D reconstruction. The system could operate normally and achieved the target function. The experimental results for the lumbar spine model showed that the scanning system could effectively collect the images of the model, visualize the 3D reconstruction, realize a full-dimensional stereoscopic display and provide a reference basis for lumbar puncture surgery.

Acknowledgments

This study was supported by grants from the CAMS Innovation Fund for Medical Science (No. 2021-I2M-1-042, 2021-I2M-1-015, No. 2017-I2M-1-016).

Conflict of Interest

The authors declare that there is no conflict of interest.