DeltaVLAD: An efficient optimization algorithm to discriminate speaker embedding for text-independent speaker verification

1.   Introduction

Speaker recognition (SR) is a key technology for intelligent interaction. In view of free accessibility and improved arithmetic capability of large databases such as the VoxCeleb ^[1,2], SR has made good progress in recent years. Voiceprint is a behavioral feature with physiological characteristics, unlike speech recognition, speaker recognition takes no account of the meaning of words, and it just focuses more on the speaker's identity.

Speaker recognition (SR) is mainly divided into speaker verification (SV) and speaker identification (SI) ^[3]. SV is a 1:1 task to confirm whether two voices originate from the same person, while SI is a 1:n task to confirm whether the target speaker's voice appears in the given samples. Meanwhile, SR can be divided into text-dependent and text-independent depending on the content ^[4], between which the former is more common and more challenging in the real world. We must specify that the scope of research is text-independent in this paper.

The Gaussian mixture model-universal background model (GMM-UBM) made the significant contribution to the development of SR technology from the laboratory to the application ^[5], but GMM-UBM was limited in the ability to handle complex data of a real scene, therefore we will discuss the model based on deep learning in this paper. At present, SV based on deep learning are of three branches: DNN/i-vector-based model ^[6], embedding-based model ^[7] and end-to-end model ^[8].

Due to the further development of deep learning and the strong data-fitting ability of deep learning models, the voiceprint research has gradually shifted to deep learning for feature extraction. The input of end-to-end model is a waveform with the model requiring performing a large number of calculations, making the training harder, and the embedding-based speaker recognition is more common. This paper is dedicated to research on the embedding-based speaker recognition.

In recent years, most researchers in the speaker recognition research community have focused on the feature encoding layer and have proposed many excellent feature encoding methods, such as ANF ^[9], ABP ^[10], VBA ^[11] and Segment Aggregation ^[12], which bring a great improvement in the speaker recognition models. As shown in these works, one of the contributions of this paper is to propose a new feature encoding method called Delta-VLAD. The dynamic relationship between preceding and following frames in frame-level features can also indicate the identity of the speaker, therefore, we capture such dynamic features that can indicate the identity of the speaker by calculating the delta coefficients of the context in frame-level features in Delta-VLAD. Experimental results show the EER of the method with Delta-VLAD outperforms the baseline model by 25%.

In addition to the research on feature encoding methods, the research on loss functions has received more attention in the community. In particular, the aggregation of the prototypical loss function and the margin-based softmax loss function were proposed in our previous work ^[13], which made a large performance improvement. In this paper, we keep exploring the application of the combined loss function in a SV model, and make further performance improvements by adjusting the corresponding weight $\beta$.

In summary, the main contributions of this paper are as follows:

(1) The Delta-VLAD is proposed. It captures the dynamic relationship between the preceding and following frames in frame-level features, which models the nature of dynamic features in acoustic features.

(2) A combined loss function is further explored. The combination of prototypical loss function based on the few-shot learning framework and margin-based softmax loss function is further explored in SV domain, and the advantages of such combined loss functions are fully demonstrated.

(3) The experimental results indicate that the proposed methods achieve SOTA performance on the voxceleb1 dataset.

2.   Related works

How to make full use of the voiceprint feature information is the key for SR. Generally speaking, there are three modules for optimizing SR system: frame-level feature extractor, the coding layer and loss function.

In the field of deep learning for SR, the main backbone networks are MP ^[9], DCNN ^[14], TDNN ^[7,15], and RNN ^[16]. They are used as feature extractors for extracting elements that characterize the speaker. To reduce model parameters and increase convergence speed, the fast ResNet34 network ^[17] is proposed and achieved excellent performance. Meanwhile, the attentive SE block is also used to pay more attention to the contributions of different local information.

After going through the frame-level feature extractor, the output is still frame-level features with its length related to that of the input acoustic features. The role of the encoding layer is to map the variable-length frame-level features into fixed-dimensional utterance-level embeddings for the purposes of facilitating subsequent classification and discrimination. The current methods for encoding frame level features at the pooling layer are mainly classified into statistical-based encoding methods and dictionary-based encoding methods. The statistical-based methods include AP ^[1], SP ^[7], SAP ^[17], ASP ^[18], etc. The dictionary-based encoding methods include LDE ^[17], GhostVLAD ^[19], NetVLAD ^[20], etc. Notably, NeXtVLAD ^[21] has a powerful capability in encoding feature, it can split frame-level features into multiple word spaces before aggregating and then perform VLAD operation in each subspace, significantly reducing the number of parameters as well as enhancing performance. All of them can project variable-length utterances into fixed-length speaker characterizing embeddings, but they use only the static features of speech. ^[22] shows more efficient results obtained by simply computing the pixel differences between two adjacent frames in the feature space and the corresponding PA, and focusing on simulating small displacement at the motion boundary.

The additional margin-based softmax function was introduced to the field of speaker recognition in ^[7,23], and ^[24,9,25] used a prototype loss function to train text-independent speaker recognition models, where the speaker optimized the metric space by finding the class prototype closest to the target recognition sample. ^[26,27] combined prototype loss function and softmax loss function to train a speaker recognition model, which could be used to produce more discriminative speaker feature vectors. Unlike the above cases, our loss function is a combination of metric loss function and classification loss function, in other words, a combination of a prototype loss function based on the cosine similarity and a softmax loss function based on additional margin, and both of which collaborate and complement each other and thus further augment the discrimination of speakers.

3.   Methods

In this section, we first systematically introduce the structural framework used in the work, and then specify the differential coding strategy and the combined loss function.

3.1.   System architecture

The framework structure of the SV task is shown in Figure 1.

It consists of three modules: the frame-level feature extractor, the DeltaVLAD encoding layer, and the similarity comparison module. The two audio spectrograms are passed through the same feature extractor to form two of their own frame-level features, which are fed into the coding layer for conversion to their respective utterance-level features, and the two representations are compared for similarity to determine whether they belong to the same person's voice.

(1) Frame-level feature extractor. This network consists of a fast ResNet34 network integrated squeeze-and-excitation (SE) block ^[28]. The structure of the backbone network as a feature extractor is shown in Table 1.

Arbitrary speech duration is accepted as the input, and arbitrary lengths at the frame level are generated. The Fast-ResNet34 structure is the same as the structure of the original 34-layer ResNet. However, to reduce the computational cost, only a quarter of the channels of the original model is used in each residual block of Fast-ResNet34.

(2) The coding layer. It is used to map variable-length frame-level features to fixed-length feature vectors. Inspired by the methods in literature ^[21,22], this paper proposes a new coding strategy: the DeltaVLAD technique. It extracts the first-order difference and second-order difference between features and neighboring features as the input to capture the dynamic relationship, which is one of the main contributions of this work.

(3) Loss function. The function in this subsection use the combined function, as shown in Figure 2, where the classification function can enlarge the inter-class distance and the metric function is designed to optimize the measurement space ^[29]. The combined loss function can hit the mark of optimizing the metric space and increasing the discrimination of the feature space, which is another main contribution of this work.

3.2.   DeltaVLAD encoding algorithm

The coding layer in SV is designed to encode some variable-length frame-level features and obtain a fixed-dimension utterance-level feature vector. The DeltaVLAD is a new coding method that takes full advantage of difference and NeXtVLAD.

3.2.1.   Difference

Difference is a classical mathematical tool due to simple and easy to use for many applications. Difference algorithm is also an efficient global optimization algorithm, and the formula is expressed as follows:

where, $A$ represents the order, ${W}_{L}$represents the window length of the voiceprint. The differential representation between frame levels in speaker recognition is shown as Figure 3.

When $A = 1$, the formula can be expressed as:

When $A = 2$, the formula can be expressed as:

When $A = 3$, the formula can be expressed as:

From the above equation, we can see that the differential formula is a form of weighted average difference. The first-order difference is the corresponding subtraction of the subsequent frame and the previous frame of the current frame, reflecting the dynamic relationship between two adjacent frames. The second-order difference is an expression between the previous frame and the next frame based on the first order difference. Similarly, the third-order difference is an expression between the previous frame and the next frame based on the second-order difference, which is a weighted average difference formula reflecting the relationship between the six frames adjacent to the current frame, and the current order accounts for the major proportion. The others are similar, using the differential calculation as the representation of the feature, it can well reflect the changing characteristics of speaker behavioral feature.

3.2.2.   DeltaVLAD coding layer

NeXtVLAD is an upgraded version of NetVLAD ^[20], which has significantly reduced the model parameters and improved the training efficiency with the similar performance. NeXtVLAD is originally an encoding method used for the video compression, and now introduced to the SV task.

We further use a differential strategy at the encoding layer to capture the dynamic properties of the features and encode them into low-dimensional embeddings. As shown in Figure 4, the output by the feature extractor is assumed to be ${f^l}$, then the feature input of encoding layer is spliced by first-order difference and second-order difference, thus the input of encoding layer can be expressed as follows:

Where ${\Delta ^1}$ represents first-order difference, ${\Delta ^2}$ represents second-order difference. In addition, $\{ c\}$represents the set of clustering centers. Then, the feature vector, the output of encoding layer, can be expressed as follows:

Finally, following a fully-connected layer, the representation of the speaker feature embedding can be obtained as follows:

3.3.   The proposed loss function

3.3.1.   The margin-based softmax loss function

Since the softmax loss function fails to make speaker embedding intra-class compactly and inter-class separatly, the margin-based softmax loss function has been proposed in the academic community. Compared with the softmax loss function, the AM-Softmax loss function ^[30] and AAM-Softmax loss function ^[31] are learned to expand the classification boundary and increase the feature margin between different classes by introducing a cosine margin penalty to the target logit.

The AM-Softmax loss function can be expressed as follows:

The AAM-Softmax loss function can be expressed as follows:

However, using the loss functions alone can only forces the deep features to stay apart but fails to explicitly optimize embedding space ^[29]. On the other hand, training with AM-Softmax and AAM-Softmax has been proven to be challenging for they are sensitive to the value of scale and the margin in the loss function. So we need to consider other loss functions.

3.3.2.   The prototypical loss function

The prototype loss function is derived from the prototype network ^[32]. The prototype loss function takes the similarity between instances as the input in the metric space, and its application in SV has been proven to be effective ^[9].

Suppose there are A speakers in the training set, and a mini-batch is randomly selected from the training set, including N different speakers, each with M utterances. The M-1 samples of each person form a set of support sets $S = \{ {s_{i, j}}\}$, (where $1 \leqslant i \leqslant N$, $1 \leqslant j \leqslant M - 1$), and the remaining 1 sample form a query set $Q = \{ {q_n}\}$ (where$1 \leqslant n \leqslant N$). $x_{i, j}^s$ represents the feature vector of the j_th sample of the ith speaker in the support set, and $x_n^q$represents the feature vector of the n_th speaker in the query set. The prototype of each category can be expressed as follows:

Where ${c_i}$ denotes the center of the category in the feature space of the i_th speaker.

Based on the open-set metric of speaker recognition, the similarity is defined as the distance under the cosine similarity as in literature ^[25]:

The similarity matrix ${S_{{q_n}, {c_i}}}$ represents the similarity between the query sample ${q_n}$ and the prototype center of mass ${c_i}$. Here we refer to the prototype loss function based on the cosine similarity as the cosine-prototype loss function, or CP loss function, and denote it as follows:

The model parameters can be optimized by back-propagating the gradient change of the query sample. The prototypical loss function is of strengths like simulating the few-shot learning scenario, more suitable for open-set SV tasks.

3.3.3.   The combined loss function

Being an open-set task in real scenarios, SR is featured by joint functions of classification and few-shot learning. We combine the classification loss function and the prototypical loss function to learn a metric which can make similar samples close and dissimilar ones distant in the embedding space, and consequently satisfy all the requirements of open-set SV tasks.

We combine the AM-Softmax loss function (AM-S) or the AAM-Softmax loss function (AAM-S) and the cosine-based prototype loss function (CP) ^[13]. The AM-S and AAM-S are explicitly encouraged to make the distance between classes larger, while the CP can make it possible to find the prototype close to the target sample in the speaker feature so as to optimize the metric space. Meanwhile, the CP can also handle the situation that some samples do not appear in the training set, and greatly improve the robustness of the model. The combination of two such loss functions can train a SV network model with the best results. The optimal loss function can be trained by adjusting the hyperparameters $\beta$ of the final combined loss function (take AAM-S as an example):

4.   Experiments

We perform experiments on VoxCeleb ^[1], which contains 1, 251 speakers including 1, 211 speakers in the training set and 40 speakers in the test set, a total of over 140, 000 audios for 352 hours.

4.1.   Implementation configuration

This experiment is conducted under Linux system with PyTorch framework. The experimental environment is as shown in Table 2.

The speaker features used in this experiment are 40-dimensional log Mel-Filterbank features(Fbank)with a frame length of 25 ms and a step size of 10 ms. Since the speech in the VoxCeleb-1 dataset is continuous, neither voice activity detection (VAD) is used in the experiment, and nor data augmentation operation is performed except for random sampling.

The baseline model uses fast-ResNet as the feature extractor, NetVLAD (K = 10) as the feature encoding layer, and the loss function is AAM-Softmax (m = 0.1), which is chosen to evaluate the performance of the present modules. The model in the experiment adopts Adam optimizer, whose weight decay rate is $5e - 5$, with batchsize is set to 256 and the initial learning rate 0.005, and when the loss decrease is less than 0.01 within 5 epochs, the current learning rate is halved.

4.2.   Evaluation of combined loss functions

In this experiment, we explore the best weight$\beta$in Eq (14) under the framework with DeltaVLAD layer, and set K = 10 and G = 8. The training loss function is AAMS-CP with $m = 0.35$.

As seen in the experimental results in Figure 5, the best EER is 2.64% in $\beta = 1.4$, which means a 1:1.4 combination of prototypical loss function and margin-based classification loss function is the best setting, namely$L = {L}_{CP}+1.4\bullet {L}_{AAMS}$. The simple and beautiful form conforms to the Occam's razor principle. The following experiment is conducted with $\beta$ as 1.4.

As seen in Table 3, The combination of the proposed two loss functions outperforms any other loss or combination. For example, when the coding layer is NetVLAD, AAM-S+CP outperforms all other loss functions, with an EER of 2.99%, relatively reduced by 48.8%, 19.4%, 6.3%, 19.41% and 13.58% compared with Softmax, CP, Softmax+CP, AM-S and AAM-S, respectively.

When the coding layer is NeXtVLAD, AM-S+CP outperforms all the other loss functions, with an EER of 2.73%, relatively reduced by 45.8%, 28.9%, 28.9%, 18.3% and 16.8% compared with Softmax, CP, Softmax+CP, AM-S and AAM-S, respectively. Because the AM-S and AAM-S can make the distance between classes larger, and the CP can learn a metric which can make similar samples close and dissimilar ones distant in the embedding space, In addition, when the encoding layer is DeltaVLAD, the EER of AAM-S+CP is 2.64% with a performance improvement of 3.3% compared with AM-S+CP due to its more powerful feature discrimination by maximising classification boundaries in angular space. The effectiveness and superiority of the combination of the AAM-S loss function and the CP loss function have been fully demonstrated in our system with Res+DeltaVLAD.

4.3.   Evaluation of DeltaNeXtVLAD

The model with fast-ResNet34, NeXtVLAD and AAM-Softmax is used as baseline, meanwhile, we have also compared several of the better models available today. The Table 4 below gives the results of the comparison.

As seen in Table 4, the result of the proposed encoding layer is 2.64%, clearly superior to the baseline of 3.52%. Besides, we compare several models that work well today in Table 4, the effect of our proposed encoding layers outperforms the effects of all the others. Especially, the model with the DeltaVLAD encoding layer and AAM-S+CP loss function has best performance with an EER of 2.64%, reduced by 10.2%, 18.01%, 15.65% and 24.36% compared with NeXtVLAD, FPM, ANF and SP, respectively. That is because differencing can calculate the dynamic changes between frames, which is more conducive to capturing insignificant changes and NeXtVLAD improves processing accuracy and speed. Meanwhile, the number of parameters of the model with DeltaVLAD is about 1.58M, similar to the others, and no additional parameters are added due to the simplicity of differencing.

From the above results, it can be concluded that our proposed encoding strategy and combined loss function is effective. The result of 2.64% achieves the state-of-the-art in the open-set task. DeltaVLAD's ease of use and simple mathematical theory underline its strengths.

4.4.   Ablation studies

Ablative experiments are conducted to verify the proposed DeltaVLAD encoding layer and the combined loss function, and the results are presented in the following Table 5.

As seen in Table 5, several ablation experiments are conducted to verify the random combination of different loss functions and the encoding methods on performance. Separate use of combination CP+NeXtVLAD and combination AAM-S+NeXtVLAD get an EER of 3.52% and 3.28% respectively, and both are higher than that of combination AAM-S+CP+NeXtVLAD, which is 3.18%, but the EER of the combination proposed in this paper is 2.64%, which shows superior performance to the others, which shows the effectiveness and superiority of our proposed approach. That is because the combined loss function has the advantages of both the metric loss function and the classification loss function, which can maximize the inter-class distance and reduce the intra-class distance, and the few-shot feature of CP loss function comes with a class-balanced sampling strategy, which is more suitable for open-set tasks just like speaker recognition. Besides, differential processing of the acoustic spectrum can capture unique changes in sound characteristics between the preceding and following frames, helping to extract more discriminative features.

5.   Conclusions

In this paper, we propose an embedding-based speaker recognition model for extracting more discriminative speaker feature vectors in open-set task. The model uses a classical mathematical tool to optimise coding, called DeltaVLAD and a combined loss function, called AAM-S+CP loss function, and experiments conducted verify its effectiveness. DeltaVLAD fusing difference and NeXtVLAD, where difference assists us capture the changes between the preceding and following frames of the current frame and NeXtVLAD encoding method split the frame-level features into multiple word spaces before aggregation, and then perform VLAD operations in each subspace, significantly reducing the number of parameters and improving performance. In addition, we further explored a combined AAM-S+CP loss function, whose clear boundary capabilities reduces the intra-class distance and increases the inter-class distance at the same time, making the features more discriminative. The whole system architecture is simple and generous, in line with Occam's Razor's Law. Experiments conducted on the Voxceleb1 benchmark yield excellent results and demonstrate the effectiveness of the proposed model for the SV task.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61573151 and Grant 6197609, the Science and Technology Planning Project of Guangdong Province under Grant 2018B030323026 and the Young Innovative Talents Project of Guangdong Province under Grant 2018GkQNCX005.

Conflict of interest

The authors declare no conflicts of interest in this paper.