Isolating switch state detection system based on depth information guidance

1.   Introduction

In the process of smart substation construction, automatic control can be realized based on monitoring the real-time state of each controllable power equipment ^[1,2,3]. The isolation switch is an important piece of equipment in the substation, which converts lines and isolates high-voltage equipment ^[4]. It is essential to ensure that the isolating switch is disconnected, which relates to the safety of workers in the high voltage line maintenance. However, most isolating switches operate outdoors, and the equipment can be oxidized. The aging of transmission equipment may lead to abnormal states, such as improper equipment opening and closing or blocking the isolating switch. These abnormal cases will make the equipment unable to work correctly, even resulting in safety incidents. Therefore, it is significant to detect the state of the isolation switch exactly in real time.

Currently, research on the detection of the state of isolation switches in substations is currently divided into two categories. The first category involves the use of sensors ^[5,6] to detect characteristics (e.g., voltage, current, resistance, magnetic induction ^[7]) to determine the current status of the isolation switch. Li et al. ^[8] proposed a noncontact current sensing and measurement method based on tunnel magnetoresistance sensors. This method employs highly sensitive linear sensors that utilize the resistance-changing characteristics of tunnel magnetoresistance sensors with variations in the direction and magnitude of the external magnetic field. When the transmission device drives the circuit breaker to rotate, the permanent magnet also rotates, causing a change in the magnetic field, and the resistance of the magnetoresistance sensor on one end changes accordingly, thereby achieving the detection of the isolating switch state. Yuan et al. ^[9] achieved isolating switch state detection by installing three-direction pose sensors on the transmission device of the isolating switch. The sensors are mounted on the operating level of the transmission device. When the operating lever undergoes an angle change, the pose sensors measure the rotational angle through pose changes, thereby detecting the isolating switch state.

The second category is based on computer vision. The state of isolation switches based on computer vision is efficiency and flexibility, which has become a new trend ^[10,11]. Specifically, computer vision has performed significantly superior in most tasks, such as target extraction ^[12,13], target classification ^[14], edge detection ^[15], and pose recognition ^[16]. The different lighting conditions and shooting angles increase the difficulty of isolating switch detection ^[17]. So, most works carried out edge detection and feature extraction based on the structural characteristics of the isolating switch. These works judged the state of the isolating switch through the change of position and posture during the movement of the isolating switch. Zhang et al. ^[18] used the improved Hough transform method to detect the line, identify the isolating switch, and then judge the state. The direction gradient histogram (HOG) and support vector machine (SVM) are used as detection and state judgment of the isolating switch in ^[19]. Wang et al. ^[20] realized the detection and tracking of isolating switches through feature extraction and the AdaBoost classifier. By tracking the position of the isolating switch in real time, the state of the switch is determined according to the distance between the two arms of the switch. Yang et al. ^[21] combined scale invariance with the Hough transform, and proposed an algorithm for isolating switch automatic recognition based on computer intelligent vision technology. Lu et al. ^[22], based on image geometry monitored substation switches. After obtaining the contour of the arm image, the angle of the arm is calculated by the cosine theorem and the switch state is finally determined. Bin et al. ^[23] extracted and matched the scale-invariant features of the isolating switch. They transformed the features, and their method performed well in the case of image rotation and uneven illumination. Some works make an effort to introduce extra information (e.g., depth ^[24], temperature ^[25,26,27]) to detect the state of the isolation switch comprehensively. Xu et al. ^[24] calculated the actual angle between the switch arm and the insulator by exploiting the depth map and feature information. Ullah et al. ^[26,27] utilized temperature information to classify the high-voltage equipment and detect the state of the isolation switch.

Although a lot of work for the detection of the state of isolation switch has arisen, there are still two main problems. First, most of the above methods combine prior knowledge and geometric principles to improve detection accuracy. Meanwhile, some assumptions are used to reduce the complexity of the scene. However, for the different shooting angles and lighting conditions in real scenes, isolating switches in the images are incredibly diverse in shape, appearance, and size and their backgrounds are also other. Second, these methods can detect the state of one group of isolating switches successfully. In practical application, we must meet the requirements of synchronous detection, which is required to detect the state of multiple groups of isolating switches in the area simultaneously.

To tackle the above problems, an isolating switch state detection system based on depth information guidance is developed in this paper, which can detect multiple groups of isolating switches and judge the state in real time. Specifically, first, the depth camera is used to obtain red green blue (RGB) images and depth images of the isolating switch scene. Depth information is essential to exploit three-dimensional data of the isolating switch. Second, a two-stage isolating switch state detection method is proposed. We use the saliency object detection algorithm ^[28,29,30] to obtain the position of the isolating switch, then, we use a fully-connected conditional random field (CRF) ^[31,32,33] to reconstruct a clear edge. After obtaining the switch state detection results, we use Hough transform line detection to get the longest boundary in the isolating switch area. The state of the isolating switch is judged by calculating the angle between the boundary lines of the switch arm and the distance between the arms. In summary, the main contributions of this paper are as follows.

● An automatic state detection system of isolating switch is developed, integrating a controllable pan tilt, a depth camera, and an industrial computer. The system aims to detect the state of isolating switches with high accuracy and speed.

● A two-stage method is proposed, which applies the saliency object detection method to the isolating switch state detection system first. We then use a conditional random field to optimize the edge of the detection results. This method has good accuracy performance for isolating switch segmentation.

● Our system can meet the needs of isolating switch synchronization detection and judge whether the states of multiple groups of isolating switches are the same. According to the results, whether malfunction exists on one of the isolating switches or not can be judged.

The rest of this paper is organized as follows. The Section 2 introduces the hardware structure of the system. In the same section, we elaborate on isolating switch detection and state judgment. The experimental results are presented in the Section 3. Finally, the Section 4 concludes our work.

2.   Materials and methods

The logical flowchart illustrating all parts of Section 2 is shown in Figure 1. In the hardware structure part, we will illustrate the three main components of the system and their respective functionalities. In the two-stage strategy part, we will describe how the two-stage method, composed of isolating switch segmentation and fully connected CRFs, is employed to achieve the segmentation and edge optimization of the isolating switch. Finally, the part on judgement will elucidate the process of isolating switch state judgement.

Figure 1.  Logical flowchart of the developed isolating switch state detection system.

DownLoad: Full-Size Img PowerPoint

2.1.   System hardware structure

In this paper, the investigated isolating switch is a type of horizontally rotating isolating switch, as illustrated in Figure 2. This isolating switch primarily consists of two isolating blades, with the rotational range of each isolating blade being 0–90°. When the isolating blades are in the closed position, the circuit is in a closed state, allowing current to flow. Conversely, when the isolating blades are in the open position, the circuit is disconnected, leading to the interruption of current flow.

Figure 2.  The figures (a, b) and module (c) of the isolating switch.

DownLoad: Full-Size Img PowerPoint

As shown in Figure 3, the system comprises an image acquisition module consisting of a pan tilt, a depth camera, and an industrial computer for image analysis.

Figure 3.  The developed isolating switch state detection system.

DownLoad: Full-Size Img PowerPoint

Table 1 shows the detailed information of the devices. The depth camera, also known as a 3D camera, is a sensor capable of acquiring depth information from a scene. Taking the ZED2 depth camera produced by StereoLabs Inc., as utilized in this paper, as an example, the ZED2 primarily utilizes stereo vision through binocular cameras to obtain depth information. By employing functions such as camera calibration, stereo rectification, and stereo matching, it calculates the depth information of the captured scene, enabling the acquisition of both RGB and depth images. However, the limitation of a depth camera resides in imposing a considerable load on the system hardware, demanding higher hardware specifications. The controllable pan tilt can rotate horizontally and vertically and can change the shooting angles of the camera according to the needs of information acquisition. The depth camera can collect depth and RGB images simultaneously, providing critical information for monitoring. The industrial computer is used to deploy software algorithms to process the collected images in real time and judge the state of the isolating switches. The remote server is responsible for controlling the pan tilt rotation, displaying the images collected by the camera and the processing results.

Table 1.  Hardware devices used in this work for isolation switch state detection system.

Hardware device	Model	Specification
Pan tilt	Three axis pan tilt	Horizontal axis motor, Rotation range: -90°–90°; Horizontal roller motor, maintain lateral stability; Pitch axis motor, Rotation range: -60°–60°
Depth camera	ZED2	Dimensions: 175.25 mm × 30.25 mm × 43.10 mm; Power: power via USB 5V / 380mA; Image sensors resolution: Dual 4M pixels sensors with 2-micron pixels; Depth FOV: 110° (H) × 70° (V) × 120° (D) max;
CPU	Data processing module	Operating system: Windows 10; CPU: Intel Core i7-10700F; GPU: RTX 3060-12G
Remote server	Data terminal	Operating system: Windows 10; CPU: Intel Core i9-10900KF; GPU: RTX 3090

---

 | Show Table

DownLoad: CSV

The system has the following advantages:

We innovatively combine a controllable pan tilt, a depth camera, and an industrial computer. Using the controllable pan tilt instead of the fixed bracket, it can control the pan tilt to change the monitoring perspective and realize the monitoring of multiple groups of isolating switches in different areas.

Each system has an industrial control computer that can combine multiple procedures into an internet of things network.

2.2.   Method

In this paper, we propose an automatic detection method for the states of the isolating switch in the substation. The specific implementation process of the proposed method is shown in Figure 4.

Figure 4.  The flow chart of the proposed method.

DownLoad: Full-Size Img PowerPoint

First, RGB images and depth images collected by the depth camera are regarded as the inputs of the RGB with depth (RGB-D) saliency network. We use the RGB-D saliency object detection method instead of conventional object detection. We can better separate the target from the background and similar objects with depth information guidance.

Deeper models with multiple maximum pooling layers have been proven to be very effective in object detection tasks. Still, their increased invariance and large acceptance domain make it more challenging to obtain the refined target edge. The blurred edge segmentation will make subsequent processing complex, and the measurement results have a large error.

We refine the target edge by adding a fully connected CRF to the model. Line detection is performed on the optimized segmentation results, and the angles between the corresponding isolating switches are calculated.

Finally, the state of the isolating switch is judged according to the angle range.

2.2.1.   Isolating switch detection method

The output results of common target detection methods, such as you look only once (YOLO), fast region convolutional neural networks (Fast-RCNN), and so on, are the coordinate information of the targets located. These methods cannot achieve the accurate segmentation of the target region boundary. Visual saliency refers to quickly detecting the most unique visual region by imitating the human visual system and completing the detection and boundary location of saliency objects at the same time. For the single object detection scenario, the single object training of the salient target detection network can improve the robustness of the network to specific targets.

Unlike RGB saliency object detection, the RGB-D saliency object detection model considers color and depth image information together to identify the salient object. Generally speaking, in detecting salient objects, spatial depth information can be one of the important salient features ^[34]. In addition, the depth information can effectively eliminate the interference of complex background texture, and it is easier to detect the isolating switch target in the image. In an RGB-D saliency network, color image, and depth image can improve the effect of isolating switch segmentation.

This paper uses the hierarchical depth awareness network (HiDAnet) ^[35] network for saliency object detection. As shown in Figure 5, the isolating switch segmentation network based on HiDAnet proposed a novel granularity-based attention scheme that attends to fine-grained details in order to strengthen the feature discriminability of each modality. HiDAnet includes four main modules, namely, granularity-based attention (GBA), cross dual-attention module (CDA), efficient multi-input fusion (EMI), and receptive field block (RFB ^[36]). White blocks denote the network backbone. The GBA strengthens the discriminatory power of RGB and depth features separately. The CDA module takes advantage of cross-domain cues to attentively realize multimodal and multilevel fusion in a coarse-to-fine manner. The efficient fusion scheme effectively models the shared information from each modality. The shared features are further improved with the skip connections for final saliency map generation.

Figure 5.  The overall architecture of isolating switch segmentation network.

DownLoad: Full-Size Img PowerPoint

2.2.2.   Fully connected CRFs for edge optimization

The CRF is a probability graphical model based on an undirected graph, where nodes of the graph represent random variables and edges depict relationships between these variables. The essence lies in that, for any node in the undirected graph, its conditional probability distribution is only dependent on connected nodes and is independent of other nodes. In the context of a CRF, it is divided into observable nodes $X$ and latent state nodes $Y$, interconnected in a certain pattern to represent explicit and implicit relationships between nodes.

In this paper, to fully exploit the strong correlation among image pixels, a fully connected CRF is employed, where each pixel is connected to every other pixel, forming the connectivity edges. For pixel $j$, the color vector is denoted as ${I_j}$ and the label is ${X_j}$. ${I_j}$ belongs to the set of possible pixel intensities in the input image, denoted as $I = \{ {I_1}, \cdots, {I_N}\}$, while ${X_j}$ belongs to the set of possible labels assigned to pixels, denoted as $X = \{ {X_1}, \cdots, {X_N}\}$, where $N$ represents the size of the input image. Consequently, we can assume that each point in the image follows a Gibbs distribution, as formulated in Eq (1).

where $x$ represents the observed values and $Z(I)$ is the normalization factor used to convert the final output into a probability form. $E(X|I)$ is the energy function.

Based on the number of variables, the potential functions of CRF are typically categorized into unary potentials, pairwise potentials, and higher-order potentials. Although higher-order potentials can model more extensive node information, the computational complexity of such potential functions is higher. Therefore, we use unary and pairwise potentials to describe the energy function $E(X|I)$, where the unary potential measures the category probability of a pixel, and the pairwise potential characterizes the relationships between pixels. This encourages similar pixels to be assigned the same label while discouraging significantly different pixels from being assigned the same label. The expression of the energy function $E(X|I)$ is given by Eq (2).

where the range of $i$ and $j$ are from 1 to $N$ and ${x_i}$ or ${x_j}$ represents the label assignment of a pixel.

We define the expression of the unary potential function as ${\theta _i}({x_i}) = - \log P({x_i})$, where $P({x_i})$ is the label assignment probability at pixel $i$ computed by isolating switch segmentation network.

The pairwise potential function models spatial information between each pixel and its neighborhood by considering both the label field and the observation field. During the camera capture process, due to factors such as lighting variations or noise, prominent features between regions of the same category may exhibit differences and be identified as different categories in the saliency object detection process. However, the pairwise potential incorporates the impact of spatial correlations, simulating this smoothness, and taking into account label constraints. This is beneficial for edge smoothing between adjacent regions. The pairwise potential function can be expressed as Eq (3).

where $\mu ({x_i}, {x_j})$ models the label consistency. It takes 1 when ${x_i} = {x_j}$ and is 0 otherwise, adhering to a Poisson distribution. ${k^m}$ is a​ Gaussian kernel ${k^m}({f_1}, {f_2}) = \exp (- \frac{1}{2}{({f_1}, {f_2})^T}{\Lambda ^M}({f_1}, {f_2}))$ that depends on features extracted for pixels and is weighted by linear combination weight ${\omega _m}$, and the vectors ${f_i}$ and ${f_j}$ are feature vectors for pixel $i$ and $j$ in an arbitrary feature space. The model is fully connected, so there will be a pair of items between pixels in the graph no matter how far away they are. Each kernel ${k^m}$ is characterized by a symmetric, positive-definite precision matrix ${\Lambda ^M}$, which defines its shape.

We employ a bilateral position and color term to compose the kernel function. The kernel function can be explicitly expressed as Eq (4).

where the first kernel function depends on both pixel position $p$ and pixel color intensity $I$, while the second kernel function only depends on pixel position $p$. The hyperparameters ${\theta _\alpha }$, ${\theta _\beta }$, and ${\theta _\gamma }$ control the scale of the Gaussian kernels.

The sensitivity of edge contrast can be improved by using pairwise potential for edge optimization of the isolating switch segmentation results.

2.2.3.   Isolating switch state judgement

After the edge optimization of a conditional random field, the edge of the isolating switch area is more precise. We extract the direction vector representing the direction of the switch arm and calculate the angle between the corresponding switch arms. We judge the state of the isolating switch at this time according to the angle. The specific steps are as follows.

● Hough line detection is performed on each connected domain, then we select the longest line and calculate the direction vector.

● Because the output result of the saliency object detection method is an independent object region, there is no direct matching relationship between each switch arm region. According to the distance between the left points of the connected regions, the two regions with the shortest distance are selected as the matching regions.

● We calculate the cosine value of the direction vector of the two corresponding regions and the angle formed by the edges of the two switch arms.

● After many observations, if the angle is between 0~7° or 173~180° and the pixel distance between the two regions is less than the set threshold, the isolating switch is considered closed; otherwise, the isolating switch is open. The pixel distance of the area is taken as the judgment factor because the rotation range of the switch arm is 0~90°. When the isolating switch is fully opened, the two switch arms will be parallel. Therefore, when they separate, the distance should increase. Because the pixel distance is not accurate enough, it is not enough to judge the state only based on the pixel distance. The distance can only be used as an auxiliary judgment condition. Meanwhile, shooting from different angles will significantly change the threshold of pixel distance.

3.   Results and discussion

This section conducts extensive experiments to prove the feasibility of the developed isolating switch state detection system. All experiments are carried out on the Windows 10 operating system, which adopts an Intel Core i7-10700F@2.9GHz central processing unit (CPU).

3.1.   Dataset

Isolating switch segmentation networks requires substantial support from datasets during the training phase to achieve effective segmentation of isolating switches. Currently, there is a lack of publicly available RGB-D image datasets specifically tailored for isolating switch scenes. Without sufficient dataset support, neural network models may lack generalization capabilities for isolating switch scenes, and their segmentation performance may struggle to meet engineering requirements. To address this issue, this study has constructed an RGB-D image isolating switch segmentation dataset for isolating switch scenes in substations, which is utilized for training and testing neural network models.

Generally, isolating switches used in substations consists of horizontally unfolding iron double arms. When the isolating switch is fully closed, the iron double arms remain horizontal and parallel to each other with their inner contacts in contact, forming a straight line. During the opening process of the isolating switch, the iron double arms rotate horizontally outward. Isolating switches in substations are typically situated outdoors with the sky predominantly composing the background. Additionally, the isolating switch is mounted on an iron frame, connected and fixed to the iron support by cylindrical insulators. Typically, three sets of isolating switches are installed in parallel, several meters above the ground.

To obtain a neural network model with optimal segmentation performance, this section recorded multiple video segments of isolating switch scenes at the experimental site supervised by the Changzhou Power Company, utilizing a depth camera. To diversify the samples, video recording was conducted from various perspectives to obtain isolating switch images from different angles. OpenCV library, produced by OpenCV team, was employed to extract frames from the videos, capturing images from different perspectives. Each RGB and depth image has a resolution of 2208 * 1242 pixels. To prevent an abundance of highly similar images, frame extraction from the video adopted an interval frame extraction strategy. Moreover, frames with high blur caused by camera movement were manually excluded.

The dataset was annotated with ground truth images using the Labelme tool. Each single-arm isolating switch's boundary was delineated using the straight-line option in Labelme, and the arm area was filled in white while other areas were set to black.

Given the substantial similarity in video content, overfitting is prone to occur during network training. Therefore, data augmentation was applied to the existing isolating switch dataset to further expand its capacity, enhancing the generalization capabilities of the network model as much as possible. Various methods, including image mirroring, noise generation, image rotation, and image scaling, were employed to augment the existing dataset, expanding it to five times its original size. The augmented dataset reached a total of 15,000 images. The existing dataset was randomly divided into 10,000 training images, 2500 validation images, and 2500 test images, in a ratio of 4:1:1.

3.2.   Performance of the isolating switch detection algorithm

In this section, we use a depth camera to obtain RGB images and depth images of isolating switches in real-world substation scenes to evaluate the effectiveness and performance of the detection algorithm.

The detection results of this algorithm are shown in Figure 6. The experimental results show that the RGB-D saliency object detection method HiDAnet used in this paper can detect all the isolating switches in the scene. In order to observe the final segmentation and optimization results more clearly, we cut the edge region of the resulting image. After optimization of the edge of the conditional random field, better isolating switch segmentation results can be obtained. The output results show that a clearer segmentation effect can be achieved for isolating switches at different angles and the edge is more precise. When the isolating switch is at a long distance, the isolating switch segmentation result will worsen. Although it has been significantly improved after the optimization of the conditional random field, it is still not as good as that at a short distance.

Figure 6.  Detection results of the isolating switches.

DownLoad: Full-Size Img PowerPoint

In order to further verify the effectiveness and reliability of this method, as shown in Table 2 and Figure 7, we use the same dataset to compare other target segmentation algorithms. To show the detection results of the algorithm more clearly, we cut the original image to highlight the detection results of the isolating switch areas. Four evaluation metrics are used, including mean absolute error (MAE), mean F-measure, enhanced alignment measure (mean E-measure), and structure measure (S-measure) ^[37].

Table 2.  Quantitative results of proposed method and other representative SOTA methods.

Models	MAE ↓	S-measure ↑	F-measure ↑	E-measure ↑
ACNet [38]	0.3602	0.5088	0.4515	0.1904
DeepLabV3 [39]	0.4003	0.5023	0.2435	0.3382
ESANet [40]	0.3668	0.5091	0.4950	0.4954
PSANet [41]	0.3971	0.4929	0.3624	0.3944
PSPNet [42]	0.3875	0.5059	0.2743	0.3604
RedNet [43]	0.3645	0.5034	0.4348	0.4643
RefineNet [44]	0.3688	0.5139	0.4940	0.4865
UCNet [45]	0.3345	0.5144	0.6217	0.4780
STINet [46]	0.3428	0.5244	0.6045	0.4952
TSFPNet [47]	0.3157	0.5326	0.6657	0.4839
Flow-EdgeNet [48]	0.3286	0.5261	0.6347	0.4816
Ours	0.2947	0.5618	0.7162	0.5481

---

 | Show Table

DownLoad: CSV

Figure 7.  Detection results of the proposed method and other representative SOTA methods.

DownLoad: Full-Size Img PowerPoint

The MAE is a linear evaluation metric that calculates the average absolute error, pixel-wise, between saliency prediction maps and ground truth maps. A lower MAE value signifies better predictive performance. The specific formulation is as Eq (5).

where $H$ and $W$ represent the height and width of the image. $S$ and $G$ represent the saliency prediction map and the ground truth map.

The F-measure is a statistical metric in machine learning, and a higher value indicates better model performance. Prior to computing this metric, the predictive saliency maps need to be binarized using various thresholds. Subsequently, the binary saliency maps along with the ground truth maps, are employed to calculate the weighted harmonic mean of precision and recall. This process yields the corresponding F-measure. The general formulation of F-measure is Eq (6). The mean F-measure is computed by calculating the F-measure at segmentation thresholds ranging from 0 to 255 and then obtaining the average.

where $precision = \frac{{TP}}{{TP + FP}}$, and $recall = \frac{{TP}}{{TP + FN}}$. $TP$ (true positive), $FP$ (false positive), and $FN$ (false negative), respectively, represent instances of correct positive predictions, incorrect positive predictions, and incorrect negative predictions, and ${\beta ^2}$ is employed to balance the importance of precision and recall.

E-measure combines local pixel-wise information with image-level average, providing a comprehensive evaluation that considers both local and global information. E-measure is a binary foreground map assessment metric that necessitates the segmentation of saliency maps into multiple binary saliency maps using various thresholds. The general formulation is typically expressed as Eq (7).

where $\phi$ represents the enhanced alignment matrix employed to capture two crucial attributes: pixel-level matching and image-level statistics.

The S-measure is specifically designed for the evaluation of nonbinary images. Previous saliency evaluation metrics were primarily based on pixel-level errors, neglecting considerations of structural similarity within salient objects. Consequently, a new evaluation metric has been proposed by integrating both object structure similarity (${S_0}$) and region structure similarity (${S_r}$). A higher S-measure indicates better predictive performance. The formulation is expressed as Eq (8).

where ${S_0}$ represents object structure similarity and ${S_r}$ represents region structure similarity. $\alpha$ is a parameter used to balance the importance of ${S_0}$ and ${S_r}$.

The test images include a single group of isolating switches and multiple groups of isolating switches. In Table 2, we quantitatively compare the performance of our approach and other state of the art (SOTA) algorithms. Our approach achieves significant S-measure, E-measure, and F-measure performance and decreases MAE. Figure 7 shows the final segmentation results of different algorithms more intuitively. It can be seen from the figure that our method is better than other image segmentation methods in the scene of isolating switch saliency object detection. For different distances and different numbers of isolating switches, the segmented image is clearer and the edge is more obvious. The isolating switch is divided into two parts. When judging the state of the isolating switch, we need to obtain the different areas of the arms clearly, especially when the isolating switch is closed and close to each other. In order to obtain the direction of the arm, the edge line of the detection area must be extracted. If the edge of the detection result is blurred, the result of line extraction will deviate. After optimizing the edge of the CRF for the saliency object detection results, we can find that the areas of different arms are separated, and the area edges are clear in Figure 7.

3.3.   Performance of the isolation switch state detection system

There are many groups of isolating switches in a substation and they are often distributed in different areas. The camera's shooting angle range is limited, and it can only shoot the isolating switches in one area at a fixed angle. However, the controllable pan tilt in the system can change the shooting angle of the camera. Therefore, we can control the pan tilt to select the appropriate shooting angles to reduce the target occlusion and maximize the number of detection objects.

To further validate the reliability of the proposed system in isolating switch state detection, as shown in Table 3, we conducted comparisons with other isolating switch state detection methods on the same dataset. We partitioned the dataset into two categories: open and closed states, using the accuracy of the final detection results as the evaluation metric. The test data included a set of isolating switch images as well as multiple sets of isolating switch images. In Table 3, we quantitatively compared the performance of our method with other approaches. Our method exhibited superior accuracy on both open and closed state data. This indicates that our isolating switch state detection method achieves more accurate detection.

Table 3.  Quantitative results of isolating switch state detection.

Methods	Isolating switch state	Accuracy
HOG + SVM [49]	Close	96.3%
	Open	95.7%
ISSSRNet [50]	Close	98.2%
	Open	97.5%
Ours	Close	99.4%
	Open	99.1%

---

 | Show Table

DownLoad: CSV

To comprehensively assess the effectiveness of the system in different environments, we captured images of isolating switches in foggy weather conditions when creating the dataset using the depth camera. These foggy weather scene images were included in the dataset. Due to the incorporation of depth information, the proposed method in this paper demonstrated accurate detection results even in these foggy weather conditions, as illustrated in Figure 8.

Figure 8.  RGB images, depth images, and detection results in foggy weather.

DownLoad: Full-Size Img PowerPoint

In the actual detection process, the number of display frames detected is related to the number of isolating switches in the image. With the increase of isolating switches, the number of frames detected will be less. There are three groups of knife switches in the image. When the system runs under the hardware specifications provided in this paper, namely with the Windows 10 operating system, an Intel Core i9-10900KF CPU, and an RTX 3090 graphics processing unit (GPU) produced by NVIDIA Corporation, the ZED2 camera computes and outputs depth images at a rate of 40 frames per second. The real-time response rate of the system reaches 15 frames per second. As shown in Figure 9, we choose a scene with three groups of isolating switches to test the performance of the system and record the process of isolating switches from closing to opening. Figure 9 includes original RGB images and detection results. When the isolating switch starts to open, the system can detect the state change of the isolating switch in time.

Figure 9.  Experimental results of the proposed isolation switch state detection system.

DownLoad: Full-Size Img PowerPoint

To facilitate the staff's observation and control of the system, we designed an excellent graphical user interface, as shown in Figures 10 and 11. The staff can set the time of system start and shut-down according to the monitoring requirements. People can control the rotation of pan tilt on the remote server. In this way, the camera shooting angle can be changed to realize the monitoring of isolating switches in different areas. During the inspection, the real-time monitoring results processed by the industrial computer can be displayed on the remote service control terminal. The content of the observation screen includes scene, depth scene, and detection result image. The specific information on the test results is displayed in the table.

Figure 10.  Results of isolating switch state detection system (Synchronism).

DownLoad: Full-Size Img PowerPoint

Figure 11.  Results of isolating switch state detection system (Asynchronism).

DownLoad: Full-Size Img PowerPoint

4.   Conclusions

In this paper, a new isolating switch state detection system was introduced, which realized the detection, state judgment, and synchronization detection of isolating switch. The detection system consisted of a controllable pan tilt, a depth camera, and an industrial computer. The remote server can realize the control of the system and the comprehensive processing of detection information. First, the system obtained the isolating switch area by RGB-D saliency object detection. Second, in order to improve the accuracy of subsequent detection, we introduced the fully connected conditional random field to optimize the detection results and get the detection results with clearer edges. Finally, the edge line of the switch arm was extracted and the direction angle and distance between the cutter arms were calculated to judge the state of the isolating switch. In the experimental results, we compared the target detection method used in this paper with the SOTA methods in many aspects. The results showed that the proposed method in this paper obtained the best performance in terms of qualitative and quantitative analysis. Therefore, the system meets the accuracy requirements of isolating switch detection and state identification in practical application.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

Funding: This research was funded by the Science and Technology Project of State Grid Ningxia Electric Power Co., Ltd., grant number 5229CG230003.

Conflict of interest

The authors declare there is no conflict of interest.