There are 174 subjects, including 99 patients with MDD and 75 age-, sex-, and education-matched healthy controls (HC). Patients are recruited from Beijing Anding Hospital Affiliated with Capital Medical University, and the HC group is recruited through newspaper advertisements. All the patients in MDD met the DSM-IV diagnostic criteria of depression, and all the HC were interviewed using the non-patient edition of DSM-IV. Before the experiment, all of the subjects signed informed consent. The clinical characteristics of MDD and HC are shown in Table 1. p-value stands for the two-sample t-test of MDD and HC, HAMD denotes the Hanilton depression rating scale, and HAMA expresses for the Hanilton anxiety rating scale. The data we used and the data Zheng et al. [25] used were collected from the same group of subjects.

Using SPM software (version 12, University of London, London, UK), two-sample t-tests (p = 0.05) were performed on brain sMRI of 99 MDD patients and 75 HC normal. The significance level of tissue voxel values difference can be observed. Therefore, MDD’s lesion areas can be obtained, and the disease reasons can be explored. To display the lesion areas more intuitively, all images were activated. The lesion areas were stratified, as shown in Fig. 1. The red parts meant that large changes in brain structures happened between MDD patients and HC.

Fig. 1.

Activation slices in the whole brain. By analysis of FDR (false discovery rate), it can be found that the common brain lesion areas of the MDD patients where the changes of brain structure are large include the Superior frontal gyrus, dorsolateral (SFGdor), Middle temporal gyrus (MTG), Middle frontal gyrus (MFG), Postcentral gyrus (PoCG), inferior temporal gyrus (ITG), Precuneus (PCUN), Precentral gyrus (PreCG), Middle occipital gyrus (MOG), Temporal pole: superior temporal gyrus (TPOsup) and Superior frontal gyrus, medial (SFGmed) according to the extent of pathological injury of brain structures.

All the sMRI scans were acquired using a MAGNETOM Trio, A TimSystem3.0-Tesla scanner (Siemens, Erlangen, Germany) in the National Key Laboratory for Cognitive Neuroscience and Learning, Beijing Normal University, using magnetization prepared rapid gradient echo (MPRAGE). The scanning parameters are as follows: repetition time (TR) = 2530 ms, echo time (TE) = 3.39 ms, flip angle (FA) = 7${}^{\circ }$, field of view (FOV) = 256 mm $\times$ 256 mm, voxel size = 1 mm $\times$ 1 mm $\times$ 1.33 mm, slice thickness = 1.33 mm, slices number = 128.

The data preprocessing is realized using SPM12¹ (¹Available: https://www.fil.ion.ucl.ac.uk/spm/software/spm12.) toolkit based on MATLAB R2013b. Considering the important influence of the gray matter area on the diagnosis of MDD [26], only the gray matter (GM) part is used for the next experiments.

The specific preprocessing steps are shown in Fig. 2. The size of each subject’s sMRI data after preprocessing is 121 $\times$ 145 $\times$ 121 pixels.

Fig. 2.

Data preprocessing flowchart. The interference of non-brain tissue was removed by overall cleaning. GM segmentation is registered by non-linear warping to Montreal neurological institute (MNI) template generated using diffeomorphic anatomical registration through exponentiated lie algebra (DARTEL) which can obtain a high-dimensional normalization including 60 mm full bias width at half maximum (FWHM) cut-off, warping regularization of 4, spatial-adaptive non-local means (SANLM) denoising filter, and Markov random field (MRF) weighting of 0.15. The voxel size of sMRI data after normalization is 1.5 $\times$ 1.5 $\times$ 1.5. A Gaussian kernel smoothed all the data with the size 8 $\times$ 8 $\times$ 8 mm and 3 FWHM.

Although 2D DenseNet has achieved remarkable results on many 2D natural image datasets, it has few achievements in medical image analysis. The reason is that the convolution kernel and pooling kernel in 2D networks like DenseNet are two-dimensional matrices, which can only move in two directions of image height and width of the 2D images. Thus, only two-dimensional features can be extracted. However, most medical image data such as sMRI are 3D data, which can only be input into 2D networks hierarchically, or one of the dimensions must be regarded as channel dimension. And neither of the two methods can make good use of the spatial information between slices of the sMRI data. We added a depth dimension to filters such as convolution kernel and pooling kernel, which extended these kernels to the 3D matrix. In this way, the filters can move in all three directions of sMRI data, and the spatial information of data is fully mined. The output of each filter is also 3D data. If the size of one of the 3D convolution kernels is k $\mathrm{\times }$k $\mathrm{\times }$k $\mathrm{\times }$ channel, the number is n, and the input data size is h $\mathrm{\times }$w $\mathrm{\times }$d. Because the sMRI data used in this paper is similar to the grayscale image and the channel dimension is 1, the output size of the convolution kernel is described by

(1)$$(h-k+1)\times (w-k+1)\times (d-k+1)\times n$$

Similar methods can extend the pooling layer and batch normalization layer in DenseNet. The 3D-DenseNet is constructed, which can better extract representative features from 3D sMRI data and improve the classification accuracy of MDD-HC MRI data. A 121-layer 3D-DenseNet structure is shown in Fig. 3.

Fig. 3.

The structure of 3D-DenseNet121. 3D-DenseBlock (1) contains 6 layers. 3D-DenseBlock (2) contains 12 layers. 3D-DenseBlock (3) contains 24 layers, and 3D-DenseBlock (4) contains 16 layers.

Each of these layers includes a 1 $\times$ 1 $\times$ 1 convolutional layer, a 3 $\times$ 3 $\times$ 3 convolutional layer, two batch normalization (BN) [27] layers, and two rectified linear unit (ReLU) [28] layers. The structure of a 6-layer 3D-DenseBlock is shown in Fig. 4.

Fig. 4.

The structure of a 6-layer 3D-Dense block in which each arrow junction represents dense connectivity. For each layer, the feature maps of all previous layers are used as input of this layer, and the feature map of this layer is used as input of all subsequent layers.

The dense connectivity of each of these layers can be expressed as follows.

(2)$$x_l=H_l\left([x_0,x_1,\mathrm{\dots },x_{l-1}]\right),$$

where $x_l$ refers to the feature map received by layer $l$, and $[x_0,x_1,\mathrm{\dots },x_{l-1}]$ denotes the concatenation of the feature maps produced in layers. $0,\mathrm{\dots },l-1$$H_l(\cdot )$ is defined as a composite function of three consecutive operations including a BN, a ReLU, and a 3 $\times$ 3 $\times$ 3 convolution (3D-Conv). If each $H_l(\cdot )$ produces k feature maps, the total number of input feature maps of the l-th layer is $k_0+k\times (l-1)$, where $k_0$ represents the number of channels in the input layer.

The dense connection operation in Eqn. 2 is not feasible when the size of the feature maps is inconsistent, so a 3D-Transition module is added between each 3D dense block. Each 3D-Transition module contains a BN layer, a ReLU layer, a 1 $\times$ 1 $\times$ 1 convolutional layer, and an average pooling layer (AvgPool) for reducing the dimension of the feature maps. The last 3D-Dense Block is connected with a ReLU layer, an AvgPool layer, a fully connected layer (FC), and a Softmax layer for implementing the final feature reduction and classification. The specific parameters and architecture of a 121-layer 3D-DenseNet are shown in Table 2, in which each conv represents a BN-ReLU-Conv sequence, and ($\divideontimes$) denotes different part of 3D-DenseNet as shown in Fig. 3,5.

Fig. 5.

The processing workflow of the proposed transfer learning model. The proposed transfer learning method includes the following four steps. Firstly, appropriate sMRI data from the ADNI database is selected, including Alzheimer’s disease (AD), mild cognitive impairment (MCI), and healthy control (HC), a total of 656 subjects. Secondly, these data are preprocessed using the same preprocessing steps as in section 2.3. Then a 3D-DenseNet is trained with the preprocessed ADNI dataset to let the network learn the features of the sMRI data. Finally, the trained network’s backbone (the red box part) is transferred to the classification task of MDD sMRI data, and a classification layer (includes a ReLU, a 3D-AvgPool, an FC, and a Softmax) is added.

In the medical field, the amount of data is often limited, leading to bad results. The motivation of using transfer learning is to train a model with a relatively large 3D medical dataset, which can be used as the backbone pre-trained model to boost the target task with insufficient training data. In this way, we can mine more knowledge and information of the small sample data by using the related other data and transfer learning. Inspired by Chen et al. [24], we designed a novel transfer learning framework for 3D sMRI data. When it comes to the data selection, only the same part (brain) and the same type (sMRI) are collected for pre-training, and only classification tasks are considered. Because it is particularly challenging to get MDD and HC data from hospitals or labs due to privacy and there’s no open-source MDD-HC dataset on the internet, we chose to use the Alzheimer’s disease dataset (ADNI, https://ida.loni.usc.edu) as the pre-training dataset. A four-step processing workflow is designed to achieve our transfer learning model, as shown in Fig. 5.

The reason why we only select data from brain sMRI datasets for classification is that if the similarity between the selected source domain and the target domain is too small, it is likely to cause negative transfer, which will lead to worse performance, i.e., not increase but decrease of classification accuracy rate. On the contrary, the more similar the two data sets are, the more similar the high-level features of the two datasets will be, which will result in better representative features and a more suitable pre-training model for the target domain to improve classification performance. A 3D-ResNet is also trained with the same process and the same data in the third step for doing a contrast experiment. We use a small learning rate to fine-tune the backbone and a relatively large learning rate to train the classification layer. The transferred network extracts new features from our MDD data and boosts the classification performance.

The classification in this paper is a binary classification problem, that is, samples are divided into two categories, including MDD patients and HC. We specify that MDD patients are considered as positive and HC as negative. So the classification algorithm has the right or wrong predictions for the test data set, including the prediction of positive classes as positive ones (true positive, TP), the prediction of positive classes as negative ones (false negative, FN), the prediction of negative classes as positive ones (false positive, FP), and the prediction of negative classes as negative ones (true negative, TN). We select accuracy and recall as metrics to evaluate the model’s classification performance. The accuracy rate is defined as Accuracy = (TP + TN)/(TP + FN + FP + TN), which reflects the ability of the classifier to judge all samples. The recall rate is defined as Recall = TP/(TP + FN), reflecting the proportion of MDD patients correctly judged in the total number of patients. The AUC is defined as the value of the area under the receiver operating characteristic (ROC) curve.

All the networks are trained using the Adam optimization algorithm [29] with a weight decay of 0.001 and cross-entropy loss function. All the data is divided into training-validation-test sets according to the 80%–10%–10% ratio, and 5-fold cross-validation is used for 100 epochs. The data are randomly selected according to the proportion of MDD:HC in the original data set. Due to the limited memory capacity of GPU, the batch size is set to 64 when training 2D networks and to 8 when training 3D networks. When transfer learning is not used, the learning rate is set to 0.01 initially. When transfer learning is used, the initial learning rate of the non-transferred part remains the original parameter 0.01, and the learning rate for the transferred part is 0.001 times that of the original one. The learning rate will be lowered 10 times when the loss value of the validation set does not decrease for 10 consecutive epochs. All training is performed on a server with an NVIDIA TITAN Xp GPU.

During the 2D network experiments, traditional 2D DenseNet [15] is compared with 2D AlexNet [7], 2D VGG [13], and 2D ResNet [14]. The preprocessed sMRI data were hierarchically inputted into the network with an input size of 121 $\times$ 145, and a voting algorithm was used. That is, for each subject, if more than half of its layers’ test results are positive, it will be determined as a positive class. Otherwise, it will be judged as a negative class. The experimental results are shown in Table 3.

It is observed that with the increase of network layers, the classification accuracy and recall rate of the networks increase gradually, which shows that the deepening of the network can provide better non-linear expression ability, can enable the network to learn more complex knowledge, and can fit more complex input feature. Also, DenseNet performs better than other convolutional networks such as ResNet when the number of layers is approximately the same, which shows that Densenet’s dense connection idea is better than ResNet’s residual learning idea in this task. Therefore, the subsequent experiments are mainly based on DenseNet.

For proving the superiority of the 3D network, our proposed 3D DenseNet is compared with 2D DensNet [15], 2D ResNet [14], 3D ResNet [18] with different layers, and the channel dimension method. We also compare our method with some traditional machine learning methods like local binary pattern (LBP) combined with support vector machine SVM (LBP + SVM) method in which the neighbor is 8 and radius is 1, and radial basis kernel function is selected. The experimental results are shown in Table 4.

From the data in Table 4, it can be seen that the classification accuracy, the recall rate and the AUC have been significantly improved after the network is expanded to 3D (e.g., the classification accuracy of 3D-DenseNet264 is 77.42% which is higher than that of DenseNet264 69.96%). And 3D-DenseNet with a similar number of layers performs better than 3D-ResNet (e.g., the classification accuracy of 3D-DenseNet201 is 76.53%, and that of 3D-ResNet200 is 74.81%). This result indicates that the hierarchical information of MDD-MRI data is very rich, and the 3D network can mine this information effectively and provide more useful features than the 2D network. Therefore, the classification performance is improved. In addition, experimental results on Accuracy, Recall and AUC show that our proposed deep learning method can mine more rich, robust and complete features on data. It is superior to the channel dimension and traditional machine learning methods such as LBP + SVM.

For proving the positive role of transfer learning, we pre-trained the best performance on our 3D DenseNet264 model using the ADNI database and performed transfer learning (denoted by ADNI-Transfer) compared with training from scratch with MDD data, i.e., no transfer learning involved (denoted by None). Because the transfer learning method used by Chen et al. [24] (denoted by Med3D-Transfer) has only been performed on 3D ResNet series networks, and only the pre-trained model is opened. At the same time, training data cannot be provided. Therefore, to prove the superiority of our ADNI-Transfer method, we also performed the ADNI-Transfer on 3D ResNet200 [18] (denoted by None), denoted by ADNI-Transfer, and compared it with the 3D ResNet200 network with Med3D-Transfer (denoted by Med3D-Transfer). Please see the results of Table 5.

It can be seen from the data in Table 5 that the classification performance of the networks has been improved significantly after transfer learning is used (e.g., after 3D-DenseNet264 has undergone ADNI-Transfer, the classification accuracy has been increased by 6.95%). It proves that transfer learning can introduce knowledge from other fields into the classification task of MDD and HC sMRI data. To some extent, it can solve the problem of insufficient samples. At the same time, the efficiency of model training is speeded up, and the final generalization ability of the model is improved. Compared with the Med3D-Transfer method, our proposed ADNI-Transfer method has better performance, which indicates that the information extracted from source domain data with the same position and the same type as the target domain data is more valuable for the target task. Therefore, our method can improve the classification accuracy and recall rate of MDD and HC sMRI data.

The above results indicate that the classification performances are not good using 2D networks because these 2D methods ignore the information between sMRI layers. After the networks are extended to 3D, the classification accuracies are improved from 6.87% to 7.69%. And our proposed 3D-DenseNet achieved a very competitive accuracy of 77.42%. Compared to training from scratch, our proposed transfer learning method ADNI-Transfer improves the accuracy by 9.95%, which is also 2.83% higher than the existing Med3D-Transfer method. Consequently, we believe that transfer learning is of great significance in medical image classification due to the general lack of data. And it seems that the more similar the pre-training data to the target domain data is, the higher the improvement of classification performance is.