Heterogeneous cross-project defect prediction with multiple source projects based on transfer learning

1.   Introduction

Software defect prediction (SDP) is one of the most active research areas in software engineering which predicts defect proneness of new modules using historical defect data ^[1,2,3,4]. If software developers can perform defect prediction before software release, it can effectively help in reducing cost of development, shortening production cycle and improving software quality. So far, a number of software defect prediction methods are based on machine learning technique to build a prediction model using historical defect data in software dataset ^{[1,5,6,7,8,9,10,11]}. The defect data usually consists of software metrics and defect labels. Here, software metrics commonly used for defect prediction are complexity metrics (such as lines of code, Halstead metrics ^[12] and McCabe metrics ^[13]), object-oriented metrics ^[14] and process metrics ^[15], and defect labels are used to record the defect proneness of the software modules.

Early defect prediction methods predict defect proneness of a new project using historical defect data from the same project, which is called within-project defect prediction (WPDP) ^[16,17,18]. However, WPDP method is not applicable for a new project or a project that only has limited historical defect data due to lacking of sufficient data to build an accurate prediction model.

To address the limitation of WPDP method, researchers have proposed cross-project defect prediction (CPDP) ^{[1,5,6,7,8,9]}, which predicts defects for a new project using historical defect data from other projects. However, most existing CPDP methods are based on the assumption that source and target projects should have the same software metrics, while the metrics from different projects are usually heterogeneous in practice problems. For example, Table 1 shows the public defect datasets that are widely used in defect prediction including SOFTLAB, NASA, AEEEM and ReLink, which have 29, 37, 61 and 26 metrics respectively. Although there are some CPDP methods can handle heterogeneous source and target projects, they only use the common metrics which less exist between two projects and it is not sufficient to build a prediction model with such small number of metrics. Such as in Table 2, NASA and ReLink datasets only have 3 common metrics, and NASA and AEEEM have no common metrics. Therefore, the existing CPDP methods are not applicable for heterogeneous source and target projects.

Recently, there have researches about heterogeneous cross-project defect prediction (HCPDP) ^[10,11], which builds a prediction model using source and target projects with heterogeneous software metrics. Existing HCPDP methods predict defects using one source project or multiple source projects with the same metrics. Studies have shown that the performance of using multiple source projects for defect prediction is better than using only one source project ^[11]. But finding multiple projects with exactly the same metrics as source projects is a challenge. In this paper, we propose Heterogeneous Defect Prediction with Multiple source projects (HDPM) which can use multiple heterogeneous source projects for defect prediction and it can be an effective solution for the above issues. The main idea of HDPM is to construct a projective matrix between heterogeneous source and target projects which converts source project to target project space, and then use the classifier to predict the defect proneness of target project.

We use transfer learning to obtain projective matrix from multiple projects. Transfer learning is a method that allowed the domains, tasks, and distributions used in the training data and test data to be different ^[6]. When predicting defects in a project, there may be only a small scale of accessible data, and even no relevant data especially for a new project. In such condition, the predication would become a difficult task and many existing methods could not achieve good results. Transfer learning can complete a classification task by using heterogeneous data source in a different feature space or follow a different data distribution ^[19], so it can use the data from multiple heterogeneous source projects to predict defects in the target project. Furthermore, there is a phenomenon in software development that many projects copy the same project ^[20], which means they may share same library or documents. This makes transfer learning can be more suitable for applying to the defects predication of these projects. Specifically, the transfer learning provides a solution for converting source project to target project space, it can convert different features of heterogeneous projects to the vectors in the same length, and then those vectors can be used as input of the classifier for the prediction. Compared to existing CPDP methods, our method can utilize more data so it works well in defect prediction.

To evaluate the effectiveness of HDPM, we perform an experiment using four public datasets (SOFTLAB ^[21], NASA ^[21,22], AEEEM ^[23] and ReLink ^[24]) that contain 14 projects in total. And we evaluate our approach against CPDP methods including TNB ^[6], NN-filter ^[5], TCA+ ^[1], CCA+ ^[11] and WPDP method. The results show that HDPM can achieve better performance of defect prediction compared with existing CPDP methods, and outperforms or is comparable to WPDP. And the performance of defect prediction for using multiple source projects outperforms using one source project.

In this paper, we answer the following two research questions:

RQ1: Is our approach HDPM helpful for heterogeneous cross-project defect prediction?

RQ2: Whether using multiple heterogeneous source projects is applicable to software defect prediction?

The contributions of this paper are concluded as the following two points:

1) We for the first time introduce the transfer learning method Multiple Outlook MAPping algorithm (MOMAP) into defect prediction for making the distribution of source and target projects similar, and it makes full use of all the metrics of source and target projects.

2) We give a method to predict defects for one source project, and for the first time use multiple heterogeneous source projects for defect prediction to extend the data acquisition range based on this method, making software defect prediction can be better applied to software engineering.

The remainder of this paper is organized as follows: Section 2 reviews the related work of defect prediction. Section 3 introduces our approach HDPM. Section 4 describes our experimental setup and the results. The conclusion and future work are presented in Section 5.

2.   Related work

Software defect prediction is an active area in software engineering. There have been a number of researches on defect prediction ^{[1,5,6,7,8,9,10,11,16,17,18,25,26,27,28,29]}. Most of these methods use machine learning technique and the same project defect data which is called within-project defect prediction (WPDP). For instance, Flish et al. ^[16] used support vector machines (SVM) to predict defect-prone software modules at four NASA datasets. Andreou et al. ^[17] applied a machine learning technique called fuzzy decision trees to acquire accurate and reliable costs that involved in software development. Bettenburg et al. ^[18] designed MARS that has local consideration and is a hybrid between global and local models to improve the performance of defect prediction. However, there are many new projects that have not sufficient historical defect data to build a prediction model.

Some researchers attempt to use unsupervised defect prediction that without requiring source data. Zhong et al. ^[25] proposed a clustering and expert based method that applying k-means and Neural-Gas in defect prediction. Bishnu et al. ^[26] applied a Quad Tree to find the initial cluster centers to k-means algorithm for predicting faults in software modules. Abaei et al. ^[27] proposed self-organizing map (SOM) to build a prediction model that uses thresholds but not experts for labeling modules. Nam et al. ^[28] proposed novel approaches called CLA and CLAMI using the magnitude of metric values to label an unlabeled dataset. Zhang et al. ^[29] built a prediction model using a connectivity-based unsupervised classifier via spectral clustering. Although unsupervised defect prediction is the best method for defect prediction theoretically for it only uses own data to predict defects, there exist a limited number of researches on it. One important reason is the ability of unsupervised learning methods usually underperform supervised ones resulting in the prediction power of unsupervised classifiers is relatively low.

Other researchers propose a number of cross-project defect prediction (CPDP) or cross-company defect prediction (CCDP) methods about supervised ones ^{[1,5,6,7,8,9]}. Turban et al. ^[5] proposed the nearest neighbor (NN) filter which builds a prediction model by source modules that remove irrelevancies with target modules to avoid the high false-positive rate. Ma et al. ^[6] proposed Transfer Naï ve Bayes (TNB) which estimates the distribution of target data and transfers cross-company data information into the weights of source data, and using these weighted data to build a prediction model. Canfora et al. ^[7] proposed a novel multi-objective defect prediction approach using genetic algorithm based on a multi-objective logistic regression model to achieve a compromise between effectiveness and cost. Similar to the work by Turban, Peter et al. ^[8] proposed Peters filter that selects source data via the structure of training data set and test data. Nam et al. ^[1] applied a state-of-the-art transfer learning approach called transfer component analysis (TCA) to make feature distribution similar between source and target projects, and extended TCA to TCA+ that choosing different normalization for preprocessing to achieve the best performance. Chen et al. ^[9] proposed Double Transfer Boosting (DTB) which reshapes the distribution of cross-company data to fit within-company data using data gravitation method and eliminates negative instances in CC data using labeled WC data. However, these CPDP or CCDP methods are based on the assumption that source and target data should have the same software metrics. When it is hard to find a project that has the same metric with target project, the existing CPDP or CCDP methods cannot be used in the setting. But there are a number of projects that have heterogeneous metrics in dataset, so finding a new method for heterogeneous cross-project defect prediction becomes urgent.

So far, there are researches about heterogeneous cross-project defect prediction (HCPDP) ^[10,11]. Nam et al. ^[10] proposed heterogeneous defect prediction (HDP) which conducts metric selection and metric matching to predict defects across projects with heterogeneous metrics set. Jing et al. ^[11] proposed a unified metric representation (UMR) and applied canonical correlation analysis (CCA) to make the data distribution of source and target similar. The main differences between our approach and existing HCPDP methods are as follows. First, our approach can use multiple heterogeneous source projects to predict defect which is the first time used in software defect prediction. Second, our approach uses all the metrics of source and target projects for defect prediction that maintains the whole information of datasets.

3.   Approach

In this section, we describe details on our approach HDPM, which includes one source project HDPM and multiple heterogeneous source projects HDPM. Compared to predict defects using one source project, HDPM use multiple heterogeneous source projects to predict a target project. In our method, the multiple source projects can be completely different, and the target project not could be related to the source projects.

3.1.   Problem formulation and overview

Suppose there are Q labeled source projects and an unlabeled target project, the software metrics of source and target projects are different, that is, source and target projects are heterogeneous. The purpose of defect prediction is to use source projects to predict defect proneness of target project. Let $\left\{ X_{s}^{(k)} \right\}_{k = 1}^{Q}$ denote Q source projects where $X_{s}^{(k)} = \left\{x_{s}^{1}, x_{s}^{2}, \cdots, x_{s}^{N_{S k}}\right\}$, and $X_{t} = \left\{x_{t}^{1}, x_{t}^{2}, \cdots, x_{t}^{N_{t}}\right\}$ denote target project. Here, $x_{s}^{i}$ denotes the ${{i}^{th}}$ module in $X_{s}^{(k)}$ and $x_{s}^{t}$ denotes the ${{i}^{th}}$ module in ${{X}_{t}}$, ${{X}_{t}}$ and ${{N}_{t}}$ are the numbers of modules in $X_{s}^{(k)}$ and ${{X}_{t}}$. A module in source project can be represented as $x_{s}^{i} = \left\{m_{s}^{i 1}, m_{s}^{i 2}, \cdots, m_{s}^{i d_{s}}\right\}$ and a module in target project can be represented as $x_{t}^{i} = \left\{m_{t}^{i 1}, m_{t}^{i 2}, \cdots, m_{t}^{i d_{t}}\right\}$. Here, $m_{s}^{ij}$ and $m_{t}^{ij}$ respectively represent ${{j}^{th}}$ metric value of $x_{s}^{i}$ and $x_{t}^{i}$, ${{d}_{s}}$ and ${{d}_{t}}$ are the number of metrics in source and target projects. In general, the metrics that used in source and target projects are different, namely ${{d}_{s}}\ne {{d}_{t}}$. $Y_{s}^{(k)} = \left\{ y_{s}^{1}, y_{s}^{2}, \cdots, y_{s}^{{{N}_{sk}}} \right\}$ represents defect proneness of ${{k}^{th}}$ source project, $y_{s}^{i} = 1$ when $x_{s}^{i}$ has defects and $y_{s}^{i} = 0$ when $x_{s}^{i}$ is clean. The defect proneness of target project $Y_{t} = \left\{y_{t}^{1}, y_{t}^{2}, \cdots, y_{t}^{N_{t}}\right\}$ is obtained by our approach HDPM.

The target of HDPM is using transfer learning to obtain projective matrix for predicting defects. Figure 1 shows the three main phases of HDPM: (1) Firstly, features are extracted from the projects, and the data from source projects are preprocessed. (2) Then, we obtain the projective matrix by using the method of PCA and SVD, so that the heterogeneous source projects can be converted to target project space. (3) Finally, we obtain the normalized vectors of projects based on projective matrix, and train a classifier to predict the defect proneness of target project.

3.2.   Preprocessing

Typically, the defect data is class-unbalanced data that the number of defective modules in a project is much less than the number of clean modules. For example, the rate of defects of PC1 in NASA is only 8.65%. Mahmood et al. ^[3] proposed the performance of defect prediction is usually low when the data is unbalanced. That is, as the data becomes more and more balanced, the performance of defect prediction increases gradually. Therefore, we use random multi-sampling to preprocess source project, by increasing the number of defective modules to balance the data, so that all the information of source project can be retained.

Since the values of software metrics vary between different projects, it is necessary to use normalization to make the metric values of source and target projects in the same range. Normalization gives all metrics of data an equal weight and is useful for classification algorithms ^[30]. Similarly, Graf et al. ^[31] also confirmed that normalization can improve the performance of defect prediction. Therefore, we employ min-max normalization method to preprocess source and target projects which makes metric values range from zero to one. In other words, the minimum and maximum values of source and target projects are transformed into zero and one, respectively.

3.3.   One source project HDPM

After preprocessing, we introduce transfer learning method Multiple Outlook MAPping algorithm (MOMAP) ^[32] into defect prediction to make the distributions of source and target projects similar. By using this method, our approach constructs a projective matrix between heterogeneous source and target projects, in this way the source projects can be converted into target project space and the maximum correlations between them are established.

The mapping of source and target projects can be obtained by translation and rotation. The algorithm of our approach is described in Algorithm 1.

First, we remove the mean of each metric in source and target projects to translate the means to zero. Assume the means of each metric in source and target data construct the vectors ${{\mu }_{s}}$ and ${{\mu }_{t}}$ respectively. The following two equations are used to obtain a new representation of source and target projects after removing the means of each metrics.

Next, we obtain the projective matrix R using the principal direction of source and target projects, the rotation matching of source and target projects can be done by solving the following optimization problem:

where $\parallel \cdot {{\parallel }_{F}}$ is the Frobenius norm that is used to represent distance, $R \in R^{d_{t} \times d_{s}}$ is the projective matrix, ${{D}_{s}}\in {{R}^{{{d}_{s}}\times h}}$ and ${{D}_{t}}\in {{R}^{{{d}_{t}}\times h}}$ are the utilization matrices that are formed by the h principal directions of source and target projects, here, $\mathrm{h} \leq \min \left(d_{s}, d_{t}\right)$.

After rotation, source project processed by the projective matrix is added with the means of target project to make source project transform to target project space finally. Then we can use the converted source project and target project to train any classifier, such as k-nearest neighbors, SVM and Naï ve Bayes, for defect prediction.

In Eq (3), we use principal directions of source and target projects to obtain the projective matrix. The principal directions can be obtained by principal component analysis (PCA). The main idea of PCA is to recombine the original metrics and form a new set of unrelated metrics. The newly formed metrics can represent the main features of the original data and can better distinguish different types of modules. Therefore, it is reasonable to obtain the projective matrix through establishing a mapping relationship between the principal directions of source and target projects. In this paper, we use PCA to extract the first h principal directions of source and target projects. In addition, we changed the value of h varies between 2 and 10 in the experiment to get better results ^[33]. The principal directions of source and target projects can be obtained by the following steps: We first perform singular value decomposition (SVD) on source and target projects separately, and then select h eigenvectors corresponding to the h largest eigenvalues as the first h principal directions.

The expansion of optimization problem in Eq (3) is $\left\| R{{D}_{s}}-{{D}_{t}} \right\|_{F}^{2} = tr\left(D_{s}^{T}{{R}^{T}}R{{D}_{s}}- \right.\left. 2D_{t}^{T}R{{D}_{s}}+D_{t}^{T}{{D}_{t}} \right)$, here, $D_{s}^{T}{{R}^{T}}R{{D}_{s}}$ and $D_{t}^{T}{{D}_{t}}$ are constants. So Eq (3) can be transformed to the following problem:

where ${{D}_{s}}$ and ${{D}_{t}}$ consist of h principal directions, that is ${{D}_{s}} = \left[v_{s}^{1}, \cdots, v_{s}^{h} \right]$ and ${{D}_{t}} = \left[v_{t}^{1}, \cdots, v_{t}^{h} \right]$ where $v_{s}^{l}$ and $v_{t}^{l}\left(l = 1, \cdots h \right)$ are the ${{l}^{th}}$ principal directions of source and target projects, respectively. Equation (4) can be intuitively transformed to:

Equation (3) is equivalent to minimization of the first h principal directions angles of source and target projects, and Eq (5) in turn implies maximization of the sum of inner products between the principal directions of source and target projects. For the solutions of Eq (4) we refer Procrustes Analysis technique from ^[34]. We refer Procrustes Analysis technique for solutions of Eq (4) ^[35]. Since $\operatorname{tr}\left(D_{t}^{T}R{{D}_{s}} \right) = tr\left(R{{D}_{s}}D_{t}^{T} \right)$, Eq (4) is equivalent to:

Let $US{{V}^{T}}$ be the singular value decomposition (SVD) of ${{D}_{s}}D_{t}^{T}$. Define $Z = V^{T} R U$, then,

where ${{\sigma }_{k}}$ is the ${{k}^{th}}$ singular value of ${{D}_{s}}D_{t}^{T}$.

Since $tr\left(R{{D}_{s}}D_{t}^{T} \right)$ can reach its max value when Z = I, we can obtain the projective matrix $R = VZ{{U}^{T}}$.

3.4.   Multiple heterogeneous source projects HDPM

In this sub-section, we present the algorithm to use multiple source projects $\left\{X_{s}^{(k)}\right\}_{k = 1}^{Q}$ for defect prediction, where multiple source projects can have heterogeneous metrics. We transform all the source projects to target project space. Similar to Algorithm 1, we first process class-unbalanced data of source projects and use min-max normalization to preprocess $\left\{X_{s}^{(k)}\right\}_{k = 1}^{Q}$ and ${{X}_{t}}$. Next, we translate the means of multiple source projects and target project to zero, and then construct the utilization matrices $\left\{D_{s}^{(k)}\right\}_{k = 1}^{Q}$ of source projects and ${{D}_{t}}$ of target project. By the Eq (8), all source projects are converted to target project space.

The method of getting projective matrix ${{R}^{(k)}}$ is similar to that of R in Algorithm 1. Let $U^{(k)} S^{(k)} V^{(k) T}$ be the singular value decomposition (SVD) of $D_{s}^{(k)} D_{t}^{T}$, and the projective matrix of ${{k}^{th}}$ source project is $R^{(k)} = V^{(k)} U^{(k) T}$. The final ${{k}^{th}}$ source project can be represented as:

Finally, we combine all the source projects and use the classifier to build a prediction model to predict defect proneness of target project.

4.   Experiments

In this section, we present the experimental setup in detail to evaluate the performance of our approach. First, we introduce the benchmark datasets and evaluation methods which are commonly used in defect prediction. Then, we perform experiments of HDPM with one source project, followed by the experiment of HDPM with multiple heterogeneous source projects. Last, we would discuss the value of parameters that are used in the paper.

4.1.   Benchmark datasets

In our experiments, we examine 14 projects containing 7511 modules from four public datasets that are commonly used in defect prediction including SOFTLAB, NASA, AEEEM and ReLink. Table 3 shows the datasets we used in experiment, and a brief description on each dataset is presented as follows.

The SOFTLAB and NASA datasets were collected from a Turkish software company and numerous NASA contractors, respectively ^[21]. For the SOFTLAB dataset, we use three projects AR3, AR4 and AR5 which are embedded controller software in the PROMISE repository. We use three NASA projects CM1, MW1 and PC1 which have the same metrics in the PROMISE repository. Shepperd et al. ^[22] find that the NASA dataset contains conflict and inconsistent cases, therefore, we use NASA datasets cleaned by them in this study. There are 28 common metrics between NASA and SOFTLAB which are Halstead and McCabe’s cyclomatic complexity metrics.

The AEEEM dataset was collected by D’Ambros et al. ^[23], and contains 61 metrics that are the most number of metrics in public defect datasets. The whole metrics include source code metrics, previous-defect metrics, entropy-of-change metrics, entropy-of-source-code metrics and churn-of-source-code metrics. The AEEEM dataset has no common metrics with the other three datasets.

The ReLink dataset was collected by Wu et al. ^[24] to improve the performance of defect prediction by manually verifying and correcting the dataset to increase its quality. The three projects that are used in the paper have 26 code complexity metrics which are widely used in defect prediction. The ReLink dataset has only three common metrics with NASA and SOFTLAB including lines of code, lines of blank and lines of comment.

4.2.   Performance measure

To evaluate the performance of defect prediction, we use recall (pd), false-positive (pf) and F-measure as evaluation measures because they are commonly used in defect prediction. The measures are defined as follow: the number of defective modules that are predicted as defective (true positive, TP); the number of clean modules that are predicted as defective (false positive, FP); the number of defective modules that are predicted as clean (false negative, FN); the number of clean modules that are predicted as clean (true negative, TN). The four kinds of defect prediction outcomes can be more clearly showed in Table 4.

Recall: The ratio of modules that are correctly classified as defective to those defective modules.

False-positive: The ratio of modules that are wrongly classified as defective to those clean modules.

Precision: The ratio of modules that are correctly classified as defective to those classified as defective.

F-measure: A harmonic mean of precision and recall.

A good prediction model intends to find out defective modules as much as possible and few wrongly prediction. So a good prediction model desires to achieve high value of recall and precision, and low value of false-positive. However, there is a trade-off between recall and precision and it is difficult to compare the performance of several prediction models by using only recall or precision. Therefore, we choose F-measure which is a harmonic mean of recall and precision. The recall, false-positive and F-measure evaluation measures range from 0 to 1. Obviously, an ideal defect prediction model should have high values of recall and F-measure, and low value of false-positive.

4.3.   Experiment design

To evaluate the effectiveness of our approach, we compare HDPM with two cross-company defect prediction methods including TNB ^[6] and NN-filter ^[5], two cross-project defect prediction methods TCA+ ^[1] and CCA+ ^[11], and within-project defect prediction method WPDP. For TCA+, we use the source code provided by the author. We carefully re-implement TNB, NN-filter and CCA+ according to their papers. In these compared methods, TNB, NN-filter and TCA+ methods require the source and target projects should have the same metrics, so we use the common metrics in heterogeneous source and target projects. NN-filter attempts to select suitable training modules to construct predictors, and we select 5 nearest neighbors for each target module to construct the training set. We use K-Nearest Neighbor (KNN) as classifier in our approach, where Euclidean distance and the top 5 nearest neighbors are set as parameters of KNN.

We design the following two experiments to evaluate our approach:

1) One-to-one HDPM. We use all modules in one project as source project to conduct heterogeneous cross-project defect prediction. The source and target projects are from different datasets. For example, EQ $\to$ AR4, CM1 $\to$ JDT, PC1 $\to$ Apache, etc. The left of “ $\to$ ” represents the source project and the right of “ $\to$ ” represents the target project.

2) Multiple-to-one HDPM. We use all modules in multiple projects which are from different datasets as multiple heterogeneous source projects. We use five-fold crossvalidation method to evaluate our approach: Eighty percent data is used as source projects, while other 20% is used as target projects for evaluating the performance. The source and target projects are also heterogeneous. For example, {CM1, Apache, EQ} $\to$ AR3, {AR3, Apache, JDT} $\to$ CM1, etc.

Since our approach involves a degree of randomness that we process class-unbalanced data of source project in preprocessing, we run HDPM 100 times of each pair of source and target projects and add up these experiment results. We set a parameter threshold that will introduce in 4.4.1, if the number of experiments that predict a module to be defective is larger than the threshold, the module is predicted to be defective eventually. In 4.4.1, the result shows that our approach can get better performance of defect prediction when the value of threshold varies from 30 to 45. Therefore, in the following experiments, we choose threshold to 30 which means if more than 30 percent of the experiments predict a module to be defective, the module is predicted to be defective eventually.

4.4.   One-to-one HDPM

For one-to-one HDPM, we choose one project in a dataset as source project and one project in another dataset as target project, that is, source and target projects are heterogeneous. Since existing CCDP or CPDP methods use common metrics in heterogeneous source and target projects, HDPM can compare with these methods including TNB, NN-filter and TCA+ method when there are common metrics in source and target projects. For source and target projects with no common metrics, such as AEEEM and the other three datasets have no common metrics, HDPM can only compare with CCA+ method.

Table 5 shows the Pd and Pf values of our approach and other compared methods when source and target projects exist common metrics. Table 7 shows the Pd and Pf values of our approach and CCA+ method when source and target projects exist no common metrics. In these two tables, the numbers which are presented with boldface indicate the best results in these methods. The Pd and Pf values in these two tables are the average values calculated from 100 repeated runs. The average Pd and Pf values of HDPM are 0.65 and 0.07 in Table 5, and the average Pd and Pf values of HDPM are 0.61 and 0.09 in Table 7. From Tables 5 and 7, we can see that HDPM can obtain better Pd and Pf values compared with TNB, NN-filter and TCA+ in most prediction scenes, and is similar to CCA+. The average values of Pd and Pf are similar in Tables 5 and 7 whether there are common metrics in source and target projects.

Table 6 shows the F-measure values of HDPM, TNB, NN-filter, TCA+ and CCA+ method when source and target projects exist common metrics, and Table 8 shows the F-measure values when source and target projects exist no common metrics. Both of these tables are the average values calculated 100 repeated runs. The average F-measure value of HDPM is 0.63 in Table 6, and the average F-measure values of HDPM is 0.61 in Table 8. From Tables 6 and 8, we can see that HDPM can obtain better F-measure values in most cases compared with existing CPDP methods. The reason is that our approach uses all the metrics for defect prediction rather than just common metrics, so HDPM can retain all the information of source and target projects. When comparing Table 6 with Table 8, the performance of our approach is not affected significantly whether the source and target projects have common metrics. The reason is that we use principle directions of source and target projects to construct the projective matrix that the principle directions recombine all the metrics to make them irrelevant and can represent the main information of source and target projects, so HDPM is not affected by the number of common metrics. Above all, we can conclude that HDPM is helpful for HCPDP and can achieve better performance for one-to-one HDPM.

We perform the Wilcoxon rank-sum test ^[35] at a confidence level of 95% to statistically analyze the F-measure results in Tables 4 and 6, and the p-values are shown in the last row of Tables 4 and 6. As the p-values are all below 0.05, it illustrates that HDPM can significantly improve the existing CPDP methods.

4.5.   Multiple-to-one HDPM

For multiple-to-one HDPM, the experiments were conducted on a 32-bit window machine with Intel Core2 Duo E8400 3.0 GHz and 3.0 GB memory. We run the code on MATLAB. We select one project from SOFTLAB, NASA, AEEEM and ReLink datasets, respectively, and three of them as heterogeneous source projects and the other project as target project. Since the source projects are heterogeneous among themselves and they are also heterogeneous with target project, there are no common metrics in source and target projects, therefore, HDPM can only compare with within-project method (target→target) for multiple heterogeneous source projects.

Table 9 shows the Pd and Pf values of HDPM and WPDP method when using multiple heterogeneous source projects. Table 10 shows the F-measure values of HDPM and WPDP method. The Pd, Pf and F-measure values of these tables are the average values calculated from 100 repeated runs, and the numbers which are presented with boldface indicate the best results. The average Pd and Pf values of HDPM are 0.70 and 0.09 and the average F-measure values of HDPM is 0.71. From Tables 9 and 10, we can see that HDPM can obtain better performance of defect prediction compared with WPDP method. The p-value is 6.5 × 10^-8 which indicates HDPM makes a statistically significant improvement in comparison with WPDP method. As compared with one-to-one HDPM, the Pd and F-measure values are increased by about 10% and the Pf value is similar. The reason is that using multiple heterogeneous source projects contains more useful information and can build a more accurate prediction model than using only one source project, so defect prediction for multiple heterogeneous source projects outperforms one source project. Above all, we can conclude that using multiple heterogeneous source projects is applicable to HCPDP and extend the data acquisition range of defect prediction and make software defect prediction better applied to software engineering.

4.6.   Impact of parameters

In the experiments of our approach, there are two parameters may have an effect on the performance of defect prediction, one is threshold, the other is the number of repeated runs. In this section, we would discuss the effect of these parameters.

4.6.1.   Impact of threshold

In our experiment setup, we run HDPM 100 times and add up these experiment results. When there are more than threshold experiments predict a module to be defective, the module is predicted to be defective eventually. Here, we would like to investigate what value of threshold can achieve better performance of defect prediction. We take threshold between 10 and 70 and record the F-measure values. Figure 1 shows the boxplot of F-measure when taking different value of threshold. From Figure 2, we can see our approach can get better performance of defect prediction when the value of threshold varies from 30 to 45. That is, our approach can get better performance when 30-45% of the experiments predict a module to be defective and the module is predicted to be defective eventually. While the value of threshold is too low or too high, most of modules are predicted as defective or clean in the experiment, making the experiment results inaccuracy.

4.6.2.   Impact of different number of repeated runs

To deal with the randomness of our approach, that is using random sampling to process class-unbalanced data of source project, we repeated the experiment 100 times. Here, we would like to investigate whether the performance of our approach will be affected if we run HDPM with different number of repeated runs. We run HDPM 10–100 times in 40 pairs of heterogeneous source and target projects, and Figure 3 shows F-measure values with different number of repeated runs. We can see that the performance of HDPM is basically stable for different number of repeated runs, and the average F-measure values vary from 0.6083 to 0.6327. Thus, we can conclude that different number of repeated runs has limited impact on the performance of our approach.

4.7.   Answers to research questions

RQ1. Is our approach HDPM helpful for heterogeneous cross-project defect prediction?

From Tables 5 and 6 which source and target projects have common metrics, the Pd values of HDPM vary from 0.42 to 0.78 and the F-measure values of HDPM vary from 0.46 to 0.80. The average Pd and F-measure of HDPM are 0.65 and 0.63, respectively. From Tables 7 and 8 which source and target projects have no common metrics, the Pd and F-measure values of HDPM vary from 0.50 to 0.73 and 0.30 to 0.80. The average Pd and F-measure of HDPM are both 0.61. The p-value of Tables 4 and 6 are all below 0.05. Thus, we can conclude that our approach is helpful for heterogeneous cross-project defect prediction.

The existing CPDP methods can only use common metrics and achieve unsatisfactory results when source and target projects contain few common metrics. However, our approach makes full use of all the metrics of source and target projects, and can achieve better performance of defect prediction at most times compared with existing CPDP methods. And HDPM can obtain comparable or even better prediction results compared with within-project defect prediction. The performance of HDPM is similar whether there are common metrics in source and target projects or not. The p-values indicate HDPM can make a statistically significant improvement in comparison with other methods.

RQ2. Whether using multiple heterogeneous source projects is applicable to software defect prediction?

As we for the first time use multiple heterogeneous source projects to predict defects, we also investigate whether using multiple heterogeneous source projects is applicable to HCPDP. From Table 9, the Pd and Pf values of HDPM vary from 0.61 to 0.80 and 0.01 to 0.22, and the average Pd and Pf of HDPM are 0.70 and 0.09. From Table 10, the F-measure values of HDPM vary from 0.54 to 0.88 and the average F-measure is 0.71. We can see that our approach outperforms within-project defect prediction at most times. The average Pd and F-measure of multiple-to-one HDPM outperforms one-to-one HDPM by 11.11 and 14.52%, respectively. The p-value of Table 8 is 6.5 × 10^-8 which is less than 0.05. It indicates that the performance of defect prediction of multiple-to-one HDPM is better than one-to-one HDPM. In addition, all the experiments can get the results in several minutes, and this indicates that the time efficiency of our method is acceptable. Therefore, we can conclude that using multiple heterogeneous source projects is applicable to HCPDP and extend the data acquisition range of defect prediction and make software defect prediction better applied to software engineering.

4.8.   Threats to validity

Although the experiment achieved good results, the generalizability of our results may be limited since the scale of the dataset in the experiment is still small. However, we have analyzed 14 projects from 4 different public defect datasets (SOFTLAB, NASA, ReLink and AEEEM) containing a total of 7511 modules, and the data is from publish defect datasets which are well represented, so it can verify the effectiveness of the method at some extent. In the future, we plan to analyze even more defect data to reduce this threat, especially the data from commercial software projects.

We evaluated our approach in recall, false-positive and F-measure that are commonly used performance measures in defect prediction. However, there are other measures that can be used in defect prediction, such as area under curve (AUC), and G-means which are also comprehensive measures.

For the four compared methods TNB, NN-filter, TCA+ and CCA+, we got the program code of TCA+ from the author, and we carefully implement TNB, NN-filter and CCA+ by following their papers. It may affect the accuracy of the experiment.

5.   Conclusion

In this paper, we propose Heterogeneous Defect Prediction with Multiple source projects (HDPM) which can use multiple heterogeneous source projects for defect prediction to solve heterogeneous cross-project defect prediction (HCPDP) problem. HDPM constructs the projective matrix which can transfer source project to target project space, making the distributions of source and target projects are similar. Our approach is applicable for heterogeneous cross-project defect prediction and for the first time to use multiple heterogeneous source projects in software defect prediction, making software defect prediction better applied to software engineering. The results of our experiments show that HDPM can achieve better performance of defect prediction than existing CPDP methods, and can obtain comparable or even better prediction results compared with WPDP method. In addition, the performance of defect prediction for multiple-to-one HDPM outperforms one-to-one HDPM.

In the future, we plan to evaluate HDPM with more datasets to validate the generalization of our approach, and try other transfer learning methods and classifiers to improve the performance of defect prediction further.

Acknowledgements

The presented research is partly funded by the National Key Research and Development Program of China (2017YFB1003103).

Conflict of Interest

The authors declared that they have no conflicts of interest to this work.