Economic analysis of lithium-ion battery recycling

1.   Introduction

Nowadays, as the Industrial Internet of Things (IIoT) is racing ahead in smart cities, various industries make full use of CGC to get more economic value and greater profits ^[1,2,3]. Meanwhile, green computing advocates producing minimal carbon footprint, and energy optimization for the long-term sustainable development of the power enterprise. As the driving force of the IIoT system in the smart city, the energy industry utilizes the CGC to improve the utilization of the limited electrical energy resources and alleviate the energy crisis. As the core part of the IIoT system, the smart grid utilizes NILMI technology to deeply mine the power consumption information of the various electrical equipment and to provide efficient technical support for optimizing the net-side and intelligent management of the load-side ^[4,5,6]. Utilizing the concept of CGC, NILMI is an efficient way to improve the system performance of the smart grid by monitoring and identifying the large masses of electrical load. Meanwhile, the rapid development of machine learning, using deep learning (DL) technology to address the bottleneck problems of NILMI, has gradually become a hot topic, which can improve the performance and stability of the NILMI model ^[7,8]. However, the accuracy and the reliability of the NILMI model are still the challenges, which have hindered the development of CGC in the smart grid. The existing methods have low precision of the load recognition for the multiple sources load sampled data, which leads to the poor performance of NILMI in the energy IoT system ^[9,10]. There are several reasons: the accuracy of the online load identification is affected by the incomplete load feature representation; the stability of the model is poor in the load dynamics consumption, especially in the superimposed scenario of the multiple electrical devices ^[11,12]. To this end, we construct an algorithm framework of MCTS RL-based, and this model framework is a heuristic strategy optimization way to realize the identification and prediction of load power consumption in the online scenario, which can improve the performance of CGC in the smart grid.

By contrast, the non-intrusive approach is more competitive in the energy IoT system based on CGC ^[13,14]. They only rely on the measured value of the load waveform (current, voltage, and power signal) as the available critical for load identification. {Further enhancement of the discriminability} of the load features can improve the efficiency of decision optimization of the identification model ^[15,16]. So, plenty of literature and research have taken a lot of studies for extracting the discriminative load features of these load waveform signals; the majority of existing methodologies include the statistical optimization method, artificial neural networks (ANN) method, and DL algorithms ^[17,18,19].

At present, DL-based methods are divided into supervised learning and unsupervised learning. Supervised learning used a lot of training data set to obtain an optimal model, and to map all the input to the corresponding output ^[20,21,22]. The several common approaches include k-Nearest Neighbor (kNN) ^[23,24], artificial neural networks (ANN) ^[25], and support vector machines (SVM) ^[26]. In ^[23], a hybrid structure based on an improved weighted kNN and the overshoot multiples is employed to train the sampled data sets, and this method enhances the identification accuracy of the load sampled data sets on a small scale. A similar method is mentioned in the other literature, the combination of kNN and the similar time window (STW) algorithm is proposed, but this method still has a limited scope of application. In ^[27], the author utilizes the SVM model with Dempster-Shafer evidence theory for the load identification, but the influence of the computational complexity on model performance is not considered. In ^[28], the author exploits an improved ANN algorithm to design the parallel computing model and optimize the model performance of NILMI; finally, the result shows that the method can reduce the heavy calculation in the initial stage of the training model.

Considering the advantages of the convolutional neural networks (CNNs) for image recognition, plenty of literature have proved that the CNN model has demonstrated their advancement in load identification and energy prediction ^[29,30], especially the various recursive neural network (RNN) ^[31,32] algorithms. However, supervised learning methods still have the following problems: 1) a large number of annotated data sets are needed for training the model, which will lead to the computational burden; 2) there is the risk of over-reliance on prior knowledge, and it is also easier for a final model performance limitation; 3) samples imbalance result in the poor identification effect and the over-fitting; 4) there are the local optimization problems in online load identification.

By comparison, unsupervised learning is not limited to modeling the prior knowledge, but utilizing the directly unlabelled data sets modeling. The most commonly-used algorithms include clustering analysis (CA) ^[33,34] and Hidden Markov Model (HMM) ^[35] algorithms. These methods are attractive in training the general-purpose system on the small data sets and do not require annotation; they take advantage of the deployment capabilities of the environment to simplify the operations process, which is a very effective way of improving the performance of CGC. In ^[36], a modified fuzzy clustering algorithm is employed to achieve the load monitoring in NILMI, and the simulation results show that the method demonstrates excellent monitoring performance. In ^[37], a Markov-based model is constructed to achieve the energy consumption identification in the household; the method utilizes the cost of minimizing the computational power to improve the algorithm performance. However, unsupervised learning methods still have their limitations: when the load features space is large, the performance of recognition and prediction algorithms is not very good; the model performance is unstable, and the better the clustering effect, the lower the identification accuracy, especially in the complex combined load signal; the limited capacity of feature learning restricts the improvement of load recognition accuracy.

To address the mentioned problems, we present a novel algorithm framework based on RL, namely MCTS-CI. MCTS-CI aims to improve the stability and performance of the combined load online identification model in a self-learning way. This framework is an online optimization algorithm and can realize independent learning by using various related strategies. In the MCTS-CI model, we design a fusion DNN model to extract the load feature and utilize the recurrent neural network based on time sequence to complete the load spatio-temporal representation in the frequency domain, which can enhance the representation learning ability of the online combined load. To improve the accuracy of the online combined load identification in the self-learning way, we employ MCTS to build an optimized strategy network. An attentional mechanism is introduced into the structure of DNN, namely ATM-DNN, as the initial policy network of MCTS to replace the random selection, which can only focus on the specific feature of load and ignore the irrelevant feature information.

The highlight contributions of our research include the following aspects:

● Discriminative load features learning. Building ATM-DNN model to enhance the learning ability of the discrimination load feature, which mainly focuses on the specific information.

● Comprehensive load-shared knowledge base. Designing the knowledge base of load power facilities (LPF-KB) to facilitate the load collection and features storage, which can function as an auxiliary means to help the formation of the load feature mapping.

● Optimal decision mechanism. MCTS model is proposed to establish a strategic network and to obtain the optimal decision by self-learning way for improving the accuracy of the multi-load identification.

The rest of the four sections are arranged as follows: the related work and the systematical arrangement for the paper are presented in Section II. The algorithm framework of MCTS-CI is introduced in Section III. In Section IV, the experimental details, the criterion of performance evaluation, and the discussion of the results that compared with other methods are described. The conclusion of the article is demonstrated in Section V.

2.   Related work and systematical arrangement

This chapter mainly reviews the related studies of online load identification and then presents the applications of RL-based MCTS.

2.1.   Online load identification in NILMI

NILMI is a very effective method to improve the efficiency of load-side management in the smart grid. Especially, the applications of DL algorithms have brought more competitive effects to NILMI.

In ^[38], Hilbert Transform long short-term memory (HT-LSTM) model is used to improve the accuracy of load identification, and to reduce the fluctuations of the transient signal in the identification results. However, this method is only suitable for a small load sample space, and it will weaken the generalization of the model. To this end, a model based on data augmentation is presented to produce the synthetic load data for the target device, which is aim at the status switching of the appliance ^[39]; the results prove that using synthetic data can enhance the generalization performance of the model. Some studies concentrate on the high-quality load features learning, which determines the accuracy of load identification. In ^[31], a two-layer DL model based on the attention mechanism is employed to address the matter of focus from the load feature learning; the results indicate that the attention mechanisms are more advantageous for enhancing the model robustness, which ensures the stability of the model in the load combination scenarios. Multi-load combination identification and power forecasting are more challenging compared to one-class load. In ^[40], an improved hidden Markov model based on a Viterbi algorithm (VA) is used to achieve the multi-load identification, and the method can effectively preserve the temporal correlation of the electrical load.

Recently, the relevant literature shows that online load power prediction has gradually become a challenging research topic. In ^[41], the author utilizes a hybrid algorithm architecture of Fisher information theory and the online support vector regression (OSVR) to improve the load prediction accuracy. However, It is incredible to accurately capture the dynamic load features in the online operating mode by using the trained model on offline load data sets. In ^[42], the author uses the hidden Markov model to present an adaptive online learning method, and the method is used to update the model parameters recursively; the optimal parameters are applied to the load energy consumption prediction; the experimental result shows that the method can improve the reliability of load prediction performance. The same as a Markov decision process, another article proposes a deep RL framework to realize the energy fine-tuning and forecasting ^[43]. Experiments prove that the method can overcome the uncertainty of the load energy forecasting effectively, and optimize the design of the continuous-control model parameter in high-dimensional variables. As an improved Markov decision process, RL is gradually used for the online load identification and power consumption prediction, and to improve the efficiency of energy management ^[44,45,46].

2.2.   Monte-Carlo tree search RL-based

Commonly, the RL model is used to describe and solve the maximum returns between an agent and the environment. RL is also applied to solve a balance between the exploration and the exploitation in the online learning of load feature ^[47,48,49]. Compared to supervised learning and unsupervised learning, RL does not rely on the pre-trained model, but it is the sequential decision-making to require a continuous selection of actions, and then to achieve the maximum benefit from completing these actions as the best outcome. In this process, the learning strategies are built, and then the parameters are updated ^[50]. RL has been widely used in various fields and has produced remarkable results for NILMI.

As a heuristic search algorithm in the decision-making process, MCTS has been used as an assessment strategy for RL. For instance, AlphaGo is the classic application of MCTS ^[51]. In recent years, much literature present that MCTS can make the optimal decision-making in dealing with online dynamically changing data. In ^[58], the author utilizes a model-based RL method to realize the online scheduling of the residential microgrid without the preset prediction model, and then MCTS is used to find the optimal strategy network. In ^[59], MCTS is employed to solve the high dependence on the prior model of the hierarchical task network (HTN) planning; in the model, MCTS can help HTN to choose the optimal decomposition approach. {Although MCTS} has made excellent contributions in many fields, the application is limited in NILMI, especially in the online combined load scenes. In this paper, a novelty algorithm framework RL-based MCTS is proposed to realize the online combined load identification and prediction.

3.   Designed of MCTS-CI framework

In this section, we introduce the algorithm framework of MCTS-CI model. This model aims to realize the online combined load identification and prediction. The schematic diagram of the overall design for the proposed model is shown in Figure 1. Our design inspiration comes from AlphaGo Zero ^[51], which has the self-directed learning ability, and this means the improvement of learning ability by playing against itself. We introduce the actual process of NILMI, the design principle of the knowledge base of load power facilities, the detailed structure of ATM-DNN, and the load recognition mechanism based on MCTS-CI.

3.1.   NILMI

In the smart grid IoT system, by measuring and analyzing the real-time power consumption of the electrical equipment, NILMI can realize the actual load consumption prediction of household electricity in the short and medium term ^[52,53]. In the current study of NILMI, the load signals are divided into transient and steady-state signals ^[54]. However, the load transient signal has some limitations: the short load monitoring time will result in the incomplete acquisition of the load characteristic information; more expensive monitoring equipment is needed to maintain the load monitoring accuracy; the accuracy is not satisfactory in online load combination identification scenarios. So, in this paper, we monitor and collect the steady-state power signals of the electrical load for the online load identification.

In general, when the supply voltage meets the national standard, the steady-state current of the electrical equipment has a certain statistical law ^[55]. We suppose the steady-state load power also satisfies the load additivity criterion; the combined load power at time $t_l$ is described as the sum of load power from the individual electrical equipment:

where $P_z(X_l, t_l)\in R$ ($R$ is the real set) represents the total power of the load at time $t_l$; $X_l$ represents the status of the electrical equipment, $si\in [0, 1, 2, 3, \cdots, M]$, $si = 0$ indicates that the device has stopped and $si = 1$ indicates that the device is running in the status $X_1$; $P_n(X_{lsi}, t_l)$ represents the household appliance $n$ is running in the state $X_{lsi}$ at time $t_l$, where $n\in N$ and $N$ is the number of the household appliances; $\sigma(t_l)$ represents the measurement error and noise. The expression of the load power is as

where $V$ represents the magnitude of the voltage, $I$ represents the magnitude of the current, $\phi$ is the phase angle, and $r$ is the initial phase angle of the first harmonic component. Figure 2 reveals the load power waveform of the different household appliances and the combined load power waveform from a day of a real house.

3.2.   Design and implementation method of LPF-KB

In the paper, we first introduce the knowledge base of load power facilities (LPF-KB), and it is used to realize the collection and storage of the load sampled data sets. As an important part of the load identification model, LPF-KB is formed by the time-domain waveform of the load and the distribution of the frequency-domain image features at different times from the various household appliances. LPF-KB is the offline data sets of our model, and mainly realizes the data sharing and the critical load features management base on time series. The detailed structure design of LPF-KB is shown in Figure 3. In our paper, considering that the load type of households' electrical equipment and electricity usage are relatively stable, and then the steady-state load power signals are selected as the load signature (LS) to represent the load characteristics. We collect the power signals of the electricity load in real-time and to utilize the dynamic time-frequency conversion technology obtain the frequency-domain image of the load power signals.

In LPF-KB structure, we suppose that the known household appliances database $F(t)$, where, $F(t) = [F_1, F_2, \cdots, F_i]^T$. Each of $F_i = (f_{i, 1}, f_{i, 2}, \cdots, f_{i, N})$, where $F(t)$ represents the knowledge base of load power facilities at time $t$, $f_{i, N}$ represents the operating state of the $i{th}$ load power at time $t$, $N$ is the number of the facility operating state.

To improve the effectiveness of the load sampled data and to reduce the parameters complexity of the input of the neural network, we use the convolutional filtering (CF) method ^[56]. CF can enhance the load signal by eliminating the specific spatial frequencies. It is divided into low-pass filtering, band-pass filtering, and high-pass filtering according to the enhancement types (low, medium and, high frequency). CF method can efficiently realize the simultaneous filtering of the dynamic load signals in different frequency bands, and its filtering results are uncompressed in each frequency band, which can avoid the error caused by frequency segmentation ^[57].

From mathematical, the CF method is defined as

where $2n+1$ is the maximum value of the CF weight coefficient function. $P_\lambda$ is CF weight coefficient. The ideal frequency response function of the band-pass filter is as follow:

CF weight coefficient is described as

where, $\beta = 2f_2/SF$, $\alpha = 2f_1/SF$, and $SF$ is the sample frequency. From the above formulas, we can see that the frequency truncation errors can be avoided when both sides of $x_m$ exist $m$ sampling points. The value of $m$ will affect the accuracy and the speed of filtering. From the formula Eq (3.5), the weights are inversely proportional to $\lambda$, which shows the weights far from the center point are negligible.

3.3.   MCTS-CI model

To improve the accuracy of the online combination load identification, we propose a novel algorithm architecture RL-based, namely MCTS-CI. MCTS-CI includes a DNN structure based on an attention mechanism, namely ATM-DNN, and a self-learning module based on MCTS. MCTS-CI utilizes the ATM-DNN model to capture the combined load features information in the online scenario, which can focus on the specific features in the combined load while ignoring the unimportant ones. ATM-DNN can realize the fine-grained feature extraction of the load frequency domain and enhance the discrimination of the combined loads. After this operation, the RL agent can acquire the best strategies based on the experience gathered during the different training stages. Finally, this model generates actions $La_t$ and adjust its parameters $\{Ls_t, La_t, \varphi\}$, which is based on the feedback from the environment. The process of the agent interacting with the environment can be seen in Figure 4. The greatest advantage of MCTS-CI is that it can help the RL agents understand the more complex environments and map their states to the actions ^[60,61].

In this paper, we propose MCTS-CI model $M_s$, the initial strategy $\pi_s$, and $SP$ is the all sequence of the current state $Ls_t : \{Ls_t, La_t^k, Lr_{t+1}^k, Ls_{t+1}^k, La_{t+1}^k, \cdots, Ls_T^k\}_{k = 1}^K\backsim M_s, \pi_s$, where, $La_t^k$ represent the current action, $Lr_{t+1}^k$ represent the reward, $K$ is the number of iteration. First, MCTS is constructed based on the sampling results, and then to compute the action value function $Q(Ls_t, La_s)$, where $La_s\in A$, $A$ is the action set. The calculation expression of the action value function and its corresponding action is shown as follows:

where, $Q(Ls_t, a_s)$ is the state-action and $G_u$ represents its gains value in all complete sequences $SP$, $\gamma$ represents the attenuation factor. $G_u$ is described as

MCTS-CI is a model-independent method and it can prevent the solution of the dynamic programming from being too complicated. MCTS-CI is often used to find the best decision in a given domain by randomly sampling from the decision space and reconstructing a search tree according to the results. We employ the MCTS algorithm to realize the optimum search action for identifying the combination status of the online load ^[62]. The executing processes: according to the maximum probability value $p_s^t$ and the reward $v_s^t$ of the current node, which as the judgment basis of the optimal action $a_s$, the MCTS-CI determines the next execution action and state; the method is called each time of the search action for iteration until the threshold time or the preset number of cycles; from the starting, MCTS uses the initialization parameters $\varphi_\lambda(p_s^t, v_s^t)$, to perform the tree search, and then when the end of this search, the updated parameters $\varphi_\lambda^*(p_s^t, v_s^t)$ of MCTS is outputted, where $z_s^t$ indicates that the action with the highest probability of action $a_s$. This process can be described as $MCTS_\varphi\{\varphi | (f_s^t, p_s^t, v_s^t)\}\rightarrow MCTS_\varphi^*\{\varphi^* | (f_s^t, \pi_s^t, z_s^t)\}$, and the MCTS algorithm is presented in Algorithm 2. Algorithm 2 will be described later ($\S$3.4).

In ATM-DNN, the adjustment parameter $\lambda$ is to minimize the value of $z_s^t-v_s^t$, and to minimize the value of $p_s^t$ and $\pi_s^t$, where $p_s^t$ is the choosing probability of ATM-DNN and $\pi_s^t$ is the search probability of MCTS for the next state. In RL-MCTS, the loss function $L\_loss_s$ includes the mean squared error, the cross-entropy loss, and attention loss ($\S$3.4). The cross entropy loss is used for the load classification loss. The final loss function $L\_loss_s$ can be described as follows:

This expression includes the mean squared error, the cross entropy loss ^[51], and attention loss $loss\_A$ ($\S$3.4). The meaning of the parameter $c$ is to control the extent of the L2 weight regularization, which can effectively prevent over-fitting. $\eta$ is the weighting coefficient. In each iteration of MCTS, there are four steps to build a search tree: selection, expansion, simulation, and backtracking ^[62]. The overall design idea of MCTS is shown in Figure 5. In MCTS, each state node $Ls$ contains four parameters: $W_s$, $N_s$, $Q_s$, and $P_s$. $W_s$ is the sum of values in the next state; $N_s$ is the number of times visited; $Q_s$ is the average value of the next state; $P_s$ indicates the prior probability of selected action.

Step 1: selection. At time $t$ the load power features based on the frequency-domain $F_s^T = \sum_{t = 1}^{N}f_s^t$ are used as the current state of the root node $LS_0^t$, and $LS_0^t = \{ W_0^t(f_0^t, a_s), N_0^t(f_0^t, a_s), P_0^t(f_0^t, a_s), Q_0^t(f_0^t, a_s) \}$, where $W_0^t(f_0^t, a_s)$ represents the total value of the next state, $N_0^t(f_0^t, a_s)$ represents the number of the visit count, $P_0^t(f_0^t, a_s)$ represents the prior probability of the selected action $a_s$, and $Q_0^t(f_0^t, a_s)$ represents the mean of the next state $LS_0^t$. In our paper, Upper Confidence Bound Applied to Trees (UCT) is employed to select the node with the highest score until a node that un-extended child nodes. UCT is a function and can obtain evaluation values at different depths ^[63]. UCT algorithm is a special Monte Carlo search algorithm, in mathematics, UCT is defined as follows:

where $s_m$ is a child node of $s_0$, and we assume that $M$ steps are required to reach the leaf node, and $m < M$. $\frac{Q(s_m)}{N(s_m)}$ indicates the estimated probability of the child node $s_m$; $c$ is an explore parameters (the theoretically value is $\sqrt{2}$).

It is important to select the node with the maximum of $Q+U$ in the sub-branch of $m$, which is the next state until all the nodes have been traversed or the leaf node that does not reach the final MCTS. The detailed algorithm is described in Algorithm 1.

Step 2: expansion. At the end of the selection phase, the node $s_m$ needs to be expanded, and its unexpanded action $a_s$ is determined by using the tree selection strategy. Then a new node $s_m+1$ is created as a new node in the search tree. It gives the flow diagram of algorithm of the steps 1 and 2 in Figure 6. The UCT aims to find the least visited node to explore each time of the tree search.

Step 3: simulation. According to the original policy, from the extended position $s_m+1$, the simulation is run with the new start node until the stop condition of the simulation is reached. The process will get an initial score for the new node $s_m+1$, and the value is expressed as 1 or 0, which indicates true or false.

Step 4: back-propagation. The results of the simulation step are updated by the back-propagation operation, which is from the current iteration path to all the selected nodes. This process is that when the leaf nodes obtain the new observed return value $V_s$ and the visit times of the parent node $T_s$ through the simulation way, and then the UCT algorithm will update all internal node values on the search path by returning the results. This process can be described as

where $T_s$ indicates the visited times of the parent node and is the sum of the visits $T_s^i$ of all the children nodes. The observed return value $V_s$ is the weighted square of the observed return value $V_s^i$ of all child nodes. The back-propagation starts from the leaf node and progresses up the search path to the root node. Each iteration will expand the search tree, and the depth of the tree will increase by the sum of iterations.

3.4.   DNN-based strategy network model

In the paper, the strategy network model includes the internal policy and the external policy for building the MCTS-CI model. The internal strategy utilizes the UCT algorithm to optimize the performance of MCTS; the other one is a fusion DNN model with an attention mechanism, namely ATM-DNN, to replace the random strategy for achieving the sampling of the state sequence when the new state is not in the tree. The ATM-DNN model can better focus on the important feature of load. The overall structure of the model for the external policy can be shown in Figure 7.

From Figure 7, firstly, the time-frequency conversion technology is used to obtain the load frequency-domain image of the effective power. Secondly, the image segmentation technology based on the time series is used for the load frequency-domain image segmentation, which can form the load frame sequence. Then each frame image of the load frame sequence is the input of DNN for the load feature extraction. The attentional mechanism (ATM) is proposed to realize the fine-grained discrimination of the frequency-domain image in the salient regions of the load. In this process, ATM-DNN is presented as the extractor of the load feature to finish the load spatial feature map. In Figure 7, $f_i(w, h)$ represent the load spatial feature map, and $i$ represent the $i^{th}$ feature cube, and $w, h$ are used to find the specific eigenvalues. The sequence $a_i = \{a_1, a_2, a_3, \cdots, \}$ represent the attentional weights. The spatial feature of load can be presented as $u_i$, and its expression is as follow:

Referencing the former studies for the image recognition with the attentional mechanism, in the mathematical, the calculation process of $a_i$ includes the following formulas:

where $l_i$ denotes the state of the hidden layer in RNN at time $I$, $V_a^I$ represents the vector, $W_ff_{w, h, :}$ is the product of the load space features and weights, $W_ll_i$ is the dynamic weight, which can highlight the focusing performance of the load frame image.

And then $Sim_{a_{i, w, h}}$ is normalized, the expression is as follow:

where $Z$ is the number of the feature map, the introduction of $softmax$ can give more weight to important elements. The input and output of RNN are as follow:

The attention scoring criteria are used to measure the performance of the ATM-DNN model, and the attention loss function is described as

where $xs^p$ indicates the scores of the attention in the spatio-temporal dimension $M\times W\times H$; $ys^q$ indicates the score of the $q^{th}$ dimension, and $m\in (1, \dots, M)$, $wh\in (1, \dots, WH)$.

The overall design of MCTS-CI algorithm for NILMI is presented in Algorithm 2.

4.   Our experiment

In this section, we mainly present the specific experimental settings for our proposed model. It includes the acquisition of the experimental data and data preprocessing, the experimental evaluation, and then the results discussion. The following is the purpose of our experiment: 1) the performance verification of the representational capacity of the ATM-DNN model; 2) the performance evaluation of the MCTS-CI model for the online combination load identification.

4.1.   Datasets

The experimental datasets consist of two parts: the offline datasets and the online datasets. We use the offline datasets to train the external policy network ATM-DNN, and the online datasets are used to identify and predict the combined load. The experimental datasets mainly sample the high$\_$frequency steady-state load power signals, from the six different houses. The data collection period is three years (2018/05/01-2020/04/30). The sampling period of the load from household appliances is 20 ns. In each dataset, there are four typical household appliances: refrigerator (REF), micro-wave oven (MWO), induction cooker (IC), and water heater (WH). The specific quantity of electrical equipment is shown in Table 1.

4.2.   Baseline models

We select the most representative models as the baseline models to verify the stability and robustness of MCTS-CI. These models include the Factorial Hidden Markov model (FHMM) ^[65], KNN ^[66], CNN ^[67], and the bi-directional long short-term memory neural network (Bi-LSTM) ^[68].

FHMM: This method is a multi-chain model extension structure. It is widely used as a statistical model to deal with time series and state series problems. The method divides the state space into many layers, and each layer is equivalent to a complete Hidden Markov model. At any given moment, the observed probability depends on the current state of all layers. The method can overcome the overfitting of the model training with the increased sampling space and the incomplete classification of a model in the small sampling space.

kNN: This method determines the classification of the sampling data according to the category of the nearest one or several samples. The improved the kNN models generally include adding weight coefficients and using the averages distance k-nearest neighbor algorithm to realize the estimate of the multi-local mean vectors in the specific patterns of each group. In the combined load identification scenario, the kNN method is more suitable. CNN: This method mainly realizes the load features of that local are extracted by building the connection of the input of each neuron and the local acceptance domain of the previous layer. It can achieve the network parallel operation. By integrating the DNN architecture, the segmentation of the input features is formed in load signals to achieve the load event, classification, and sample regression in NILM.

Bi-LSTM: This method is used to solve the bidirectional time dependence of the load signal. LSTM realizes time memory function through the cell door switch and prevents gradient from disappearing.

4.3.   Experimental environment and parameter settings

To meet our experimental requirements, our experimental environment includes the hardware and software. The Pytorch 3.7 is used to achieve the design and implementation of the algorithms; the desktop PC with Windows 10 system and a NVIDA GeForce RTX 3080 Ti GPU, 128GB of RAM, and CUDA v10.2 are employed.

The core of our proposed model includes the ATM-DNN model and the MCTS algorithm. In ATM-DNN, the specific parameters set as follows: there are six blocks in the hidden layer, six filters in each block, the size and stride of the pooling layer are two; our activation function is ReLu; the dropout probability set at 0.45; the number of mini-batch is 512; Adam is as the optimizer; the learning rate set to 0.0006; the number of epochs set to 300. In the MCTS model, the self-play process will last for 20 h, implemented $10^5$ decision-makings, and 750 simulation operations are performed in each MCTS, which will present in the following subparagraph.

4.4.   Experimental data preparation and preprocessing

We select four representative household appliances for our experiment. Their work patterns range from simple to complex, especially the bidirectional dependence of the device can be reflected. Relatively diverse working modes are beneficial to verifying the performance indicators of our model. We sample the experimental datasets from the smart meter installed at the power entrance to the house, which can realize the acquisition and practical storage of the real-time data (current, voltage, and efficient power) in the various household appliances. The steady-state power signal of load in high-frequency contains the rich load characteristic information. The information can provide the discriminative load features for the combined load identification in various online scenarios.

The CF-based signal enhancement method is used to realize the data preprocessing. In the load data sampling, signal noises are caused by equipment operation errors, the overshoot, or abnormal peak of voltage or current of the signal. The data preprocessing method can solve the signal acquisition error and instability problems effectively; this method can correct the abnormal power values and ensure the top quality of the signal.

We have prepared a lot of sampled datasets to validate the performance of our model, and these datasets are from four houses (No. 1 house, No. 2 house, No. 3 house, and No. 4 house). We divide each dataset into three parts: the training dataset, the validation dataset, and the testing dataset. The first $70\%$ of the dataset sequence for the training model, the subsequent dataset sequence $15\%$ for validating the model, and the rest $15\%$ of the data sets for the testing model. In the online experimental stage, we use the collected data sets of No. 5 house and No. 6 house in real-time to further verify the model performance's competitiveness.

4.5.   Experimental evaluation criteria

We employed the most common evaluation criteria to verify the performance and stability of our model, such as Micro-$F_1$, Macro-$F_1$, Hamming loss (H-loss), and the mean absolute error (MAE). These methods have been widely used in the field of deep learning. The criteria can be defined as follow:

Micro-$F_1$ and Macro-$F_1$. Micro-$F_1$ and Macro-$F_1$ are the two strategies for calculating $F_1$-score of the multi-classification. It is assumed that $LTP_i^t$, $LFP_i^t$, $LTN_i^t$, and $LFN_i^t$ denote the true positive at time $t$, false positive, true negative, false negative of classify $i$. The precision, $Lpre_i^t$, and the recall, $Lrec_i^t$, of the class $i$ can be described as

So, the total precision and recall of all categories are presented as follows:

Macro-$F_1$ is described as follows:

$\overline{Lpre_i}$ indicates as the mean precision of the $\rm{i-th}$ class, $\overline{Lrec_i}$ indicates as the mean recall of the $\rm{i-th}$ class.

Hamming-loss: $Hamming-loss$ is employed to measure the times in the misclassification samples; namely the labels belonging to a sample are not predicted, but the labels that do not belong to the sample are predicted to belong to the sample. The smaller value, the better the model performance. The mathematical formula is described as

where $|M|$ represents the size of samples, $|L|$ represents the size of labels, $x_i$ and $y_i$ indicate the predicted value and real value, respectively.

MAE: we suppose that $y_i^t$ represents the real data, $y_i^{'t}$ is the estimated data of the appliance $i$ at time $t$. The mathematical expression is as follows:

4.6.   Experimental results and discussion

In this section, the model's learning ability and overall accuracy are verified, respectively. The various performance scores of the models are presented to evaluate the superiority of our model.

To improve the MCTS model's decision-making efficiency to ensure the decision-making process's reliability, we utilize the ATM-DNN structure to obtain the initial tree policy $p_s^t$ and the action values $v_s^t$, which instead of the randomly selected actions at the beginning of the program. The next time, the program of MCTS will self-learning for 20 h without any additional intervention, and then the complete search process will be completed. The outputted parameters of each iteration in MCTS are compared with the output results of ATM-DNN and updated ATM-DNN parameters as the policy improvement. We use the performance index $MSE$ and $Loss$ as the critical evaluation indicators to present the performance of the search process in model training. Then this convergence trend can be seen in Figure 8. $MSE$ indicates the degree of error between the actual output of MCTS and the predicted value of ATM-DNN, which can be used as the basis to adjust the parameters of the ATM-DNN and minimize the predicted value. $Loss$ is the cross­entropy losses, and the values represent the similarity of ATM-DNN selection strategy $p_s^t$ to the search probabilities $\pi_s$.

The training process of MCTS-CI includes two steps: the offline datasets of four different houses are used to train the ATM-DNN model, and then the results of the model training are the initial strategies for completing the initial selection in MCTS; the MCTS starts the search by self-learning and goes through exploration, simulation and, feedback. The process lasts about 20 h and implements about $10^5$ decision-makings. In each MCTS, 750 simulation operations are performed, meaning that making the new decisions is nearly 0.5 s. After about 7000 self-decisions, the ATM-DNN can efficiently guide the MCTS search to select stable actions. Finally, the process can realize the policy evaluation and improvement in the MCTS search.

We use the mean reward of RL in the training process to further validate the performance of the MCTS-CI. The results are shown in Figure 9. We employ the two datasets of No. 1 house and No. 2 house, respectively. The two datasets provide a large amount of the sampled load data resources in four household appliances. The validation results are shown in Figure 9. From the two curves, the intelligent agent goes through the initial selection and exploration. Then they eventually converge to the global optimum, which can prove the convergence of our method. From convergence result, the average reward is approximately 0.697 and 0.676, respectively, on the two datasets.

To explain the influence of the simulation times on model performance, we try to use the different simulation times to experiment with our model; the result of the experiment can be presented in Figure 10. We performed 50, 150, 350, 750 simulation cycles, respectively, and as the number of cycles increases, so does the rate of reward. It is because the increase of the simulation numbers can make more simulation numbers of the Monte Carlo tree, which can improve the accuracy of the evaluation of the action. So, it can ensure the optimal action and improve the effectiveness of the Monte Carlo tree search. From Figure 10, as the reward reaches a certain point, the performance will tend to stabilize.

In the model training process, we use the training time to test the computing power of the model, and the comparison of the training time in five models is shown in Figure 11. The sample size and the performances of the computing devices are the main factors in computing time. From Figure 11, our proposed model has the shortest model training time on the different sizes of the sample training sets.

4.7.   Validation of the model accuracy and performance

In this subsection, four datasets from the different houses are employed to verify the performance of FHMM, KNN, CNN, Bi-LSTM, and MCTS-CI. The validation process includes the following: 1) the comparison of the average accuracy with the five methods in training datasets; 2) the overall performance score of five different methods; 3) the comparison of the $Micro-F_1$ and $Macro-F_1$ scores on the specific household appliances datasets. We first present the overall accuracy of benchmark models and our model. The five-fold cross-validation on each dataset is used to ensure the reliability of our experiment results. Table 2 shows the accuracy and standard deviations of the results, which are presented in bold for the best results.

From Table 2, the accuracy of MCTS-CI is higher than other methods on different datasets. Significantly, the average accuracy of the MCTS-CI reaches $95.56\%$, which is higher than the second-ranked method $7.78\%$ on No. 1 house datasets. On the other datasets, the average accuracy of MCTS-CI is $89.91\%$, $90.95\%$, and $92.22\%$, which are superior to others $10.35\%$, $5.38\%$, and $6.25\%$, respectively. In comparison with CNN, in load combination identification, MCTS-CI is more competitive. The reason that different from image recognition, the load has the time-series feature, and CNN does not play to its strengths. It can illustrate that our model has advantages in online identification of the loads. In a word, this method will have a good application prospect for NILMI.

To go a step further and validate the performance of our model, we carry out the experimental tests from the overall and detailed aspects of the model, respectively. The scores from Micro-$F_1$, Macro-$F_1$, and $H-loss$ are employed to describe the experimental results.

The overall case: the experimental results of four datasets from the different houses are shown and discussed. Table 3 demonstrates the overall performance indicators of the benchmark models and our proposed model in different datasets from four houses, respectively. The numbers in the bold represent the performance is better than the others. Especially, from Table 3 Micro-$F_1$ score of our method can come up to 0.964, which is higher than the second-ranked model, 0.107. In addition, the score is 0.947, 0.946, 0.935 are higher than the second-ranked model Bi-LSTM 0.084, 0.092, 0.058, respectively. The same effect of model performance appears on Macro-$F_1$ score. From H-loss, the value of MCTS-CI is relatively minimal, indicating that the number of misclassification samples is low. The second-lowest is Bi-LSTM, indicating that the LSTM model has a significant advantage for load sampling data based on time series characteristics. CNN model can complete the feature extraction of load well, but it is not sensitive to the load combination identification. From MAE, the values of MCTS-CI are smaller than the other benchmark models, and it presents a high identification accuracy in different datasets.

The detailed case: to better demonstrate our approach's significant advantages, we use our model and four benchmark models on the various appliances datasets from the four houses to verify the models performance. Finally, Micro-$F_1$ and Macro-$F_1$ scores of the various appliances used in the four houses are obtained, which can be found in Figures 12 and 13. From Figure 12, Micro-$F_1$ scores of each appliance are shown from subgraph a to subgraph d; the scores of MCTS-CI in each of the appliances are the highest-scoring, and Bi-LSTM is the second-highest score, and then the FHMM is the lowest-scoring. The results similar to that presented in Figure 13, Macro-$F_1$ scores of MCTS-CI are best. Our proposed method excels the other methods in the score of each appliance. Especially, in subgraph Figure 12(a), Micro-$F_1$ score of the refrigerator, water heater, microwave oven, and induction cooker is up to $96.19\%$, $95.52\%$, $97.57\%$, and $94.37\%$, which is far superior to the other methods. In subgraph Figure 12(d), the highest score of Micro-$F_1$ is $97.96\%$, and it is found in the water heater with our method; the worst score appears in subgraph Figure 13(d), Macro-$F_1$, $53.23\%$, which is the score with FHMM in the refrigerator. The performance of FHMM is not good in our experiment, and the same situation also appears in KNN and CNN models. The Bi-LSTM is better than CNN and KNN because some memory gates mode characterizes it, and it can better represent the time-dependent relationships of sampling data.

Actual load identification and prediction test: In this subsection, we collect the online data from No. 5 house and No. 6 house as the validation datasets to verify our model performance in real-world applications. We observe the electricity consumption of two households and sample the load power signals from various household appliances on three random days. Figure 14 shows the power curve of the time-domain from the two different houses in three days. From Figure 14, we can identify four periods of electricity consumption, which mainly include 5:30 a.m. to 7:30 a.m, 11:00 a.m. to 1:00 p.m., 5:30 p.m. to 8:00 p.m., and 9:00 p.m. to 10:30 p.m. The frequency-domain image of the load power signals can also represent the load features, and they, as the input of DNN, can improve the accuracy of load feature extraction. In our model, the time-frequency conversion technology forms the load frequency-domain features. To ensure the validity of our data set, we use the CF method to improve the quality of the power signals and reduce the error of signal acquisition. After the data preparation and preprocessing, the datasets are inputted into our proposed model to extract the spatio-temporal features of load and identify the specific load power from all kinds of household appliances. The detailed load identification results can be presented in Figures 15 and 16.

As Figures 15 and 16 indicate, MCTS-CI shows a good recognition effect and stability in the test data set of the actual electricity consumption. Significantly, the experimental result of the water heater agrees with and refrigerator with the real sampled data in the usage scenarios of two different houses, which shows that MCTS-CI can achieve excellent recognition for the electrical appliance with high-frequency usage. The same results appear in the micro-wave oven of No. 5 house and the induction cooker of No. 6 house. For the household appliances with low-frequency usage, more data are needed to train the model. Our proposed method has outstanding practical value and better performance for correctly categorizing the online load.

5.   Conclusions

CGC is applied in the IoT system of the smart grid for realizing the rational allocation of the various resources and improve the production efficiency. NILMI is the most efficient way, can improve the performance and stability of CGC in the smart grid. In this paper, we develop a novel algorithm framework, MCTS-CI, which is a multi-stage algorithm framework. MCTS-CI involves a load data sharing platform, LPF-KB, a load spatio-temporal feature extraction mechanism based on ATM-DNN, and a set of optimized policy networks based on MCTC. LPF-KB can enhance the utilization rate of load sampled data to realize the fine-grained storage of load; ATM-DNN can improve the distinctiveness and effectiveness of the combined load features; the extraction of load spatio-temporal features can improve the learning ability of load representations; MCTS is used to achieve the optimal strategy for reducing the error rate of load identification and energy prediction in the online scenario. By comparison with four benchmark models, the experimental results prove that MCTS-CI has outstanding performance for improving the accuracy of the load combination identification and reducing the identification error rate of the unknown load.

The method can be widely applied for predicting household electricity consumption in online scenarios. {The method will as a design idea for realizing the short and medium-term load} identification with the external weather and periodic temporal features to improve the intelligent adjustment performance of the IoT system in the smart grid. In the future, we will utilize MCTS-CI to address the more complex and variable load combination scenarios.

Acknowledgments

The authors of this paper were supported by S & T Program of Hebei through grant 20310101D, and National Key R & D Program of China through grant 2021YFB1714800.

Conflict of interest

The authors declare there is no conflict of interest.