BE [38] is employed for feature extraction based on sequence composition information. It is utilized to transform protein sequences into digital vectors. With this approach, each amino acid is depicted as a 20-dimensional binary vector. For instance, alanine (‘A’) is denoted as [1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], and arginine (‘R’) by [0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]. The special character ‘X’ is distinguished as a 20-dimensional vector consisting of zeros: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]. Consequently, a protein sequence of length L generates a feature vector with dimensions of 20$\times$L.

CKSAAP [39] is a feature extraction method that relies on protein sequence composition and emphasizes the occurrence frequency of distinct pairs of amino acids within amino acid fragments (total of 20 $\times$ 20 potential combinations). These paired amino acids are separated by a specific number of intervening amino acids, denoted by the parameter ‘k’. For example, when k = 2, the method computes the occurrence frequency of all 400 amino acid pairs separated by two intervening amino acids within the fragment. Typically, the establishment of an upper limit ‘kmax’ is essential, allowing the computation of component characteristics for values ranging from k = 0 to kmax. Hence, for a protein sequence of length L, setting the kmax to four yields 5 $\times$ 400 = 2000 dimensional features.

EAAC utilizes sequence composition to capture the frequency of standard amino acid occurrences within individual peptide sequences. AAC transforms the 20 amino acids into a 20-dimensional digital vector by tallying the occurrence of specific amino acids in individual peptide sequences [40]. EAAC differs significantly from this by employing continuous sliding from the N-terminus to the C-terminus of peptide sequences, utilizing a fixed-length sliding window for computation [41]. Thus, for a protein sequence of length L, a fixed sliding window size of 5 would yield L-5+1 sliding windows. The EAAC encoding dimension is (L-5+1) $\times$ 20.

PWM [22] represents a positional feature extraction method. For both positive and negative samples, the respective relative frequencies of the 20 amino acids at specific positions are calculated, thus giving the frequency data for amino acids at each position. Hence, a protein sequence of length L yields an L-dimensional feature vector.

The Amino Acid Index (AAindex) compiles numerical indices encompassing diverse physicochemical and biochemical attributes of amino acids and their pairs [42]. A total of 531 prevalent physical and chemical properties were employed for feature extraction in this study. For each property, the average of the 20 amino acids was computed for the placeholder character ‘X’. For a protein sequence of length L, 531 physical and chemical properties were extracted per amino acid, resulting in a feature matrix with dimensions of 531$\times$L.

PSSM is a method for extracting features rooted in evolutionary information. Position Specific Iterative-Blast (PSI-BLAST) is widely employed for identifying remotely-related protein sequences. PSSM generation involves comparing the target sequence against homologous sequences through PSI-BLAST as follows [43]:

(1)

The protein sequence P, of length L, can be transformed into a feature matrix denoted as PSSMp, with the dimensions of L$\times$20. Within this matrix, P${}_{\mathrm{(}i\mathrm{\to }j\mathrm{)}}$ denotes the score attributed to mutation of the amino acid residue at the i${}^{\text{th}}$ position (i = 1, 2,…, L) in protein sequence P to the j${}^{\text{th}}$ amino acid (j = 1, 2,…, 20). Positive and negative score values denote the likelihood of occurrence for the respective mutation. Positive scores suggest a higher likelihood, whereas negative scores imply a lower likelihood.

The Null importances method [29] is a statistical approach used to assess the significance of feature importance generated by the RF model. It involves creating a distribution of importance scores for each feature under the null hypothesis, which assumes that the feature is irrelevant to the prediction outcome. The process is carried out by permuting the response vector in the dataset and then re-running the model to compute importance scores for each feature. These permutations effectively break any real association between the feature and the target, thereby providing a baseline distribution of importance scores under the assumption of no relationship.

Integration of the null importances method with the RF model allows for a more robust assessment of feature importance. After the RF model is fitted on the original dataset, the original importance score of each feature is calculated. The null importances distribution is then generated by repeatedly permuting the response vector and recalculating each feature’s importance score across multiple iterations. The original importance scores can then be compared against this null distribution to determine the significance of the observed importances. Features with the original importance score significantly higher than those observed in the null distribution are considered to be genuinely informative for the model’s predictions, thereby providing a means to identify and focus on the most relevant features for the problem at hand.

This method enhances the interpretability of the RF model by distinguishing between features that have a statistically significant impact on the model’s predictions and those that do not. It also reduces the risk of over-interpreting importance scores derived from random noise in the data, thereby resulting in more reliable and transparent machine learning models.

Olivier implemented this method within the Kaggle Community (https://www.kaggle.com/code/ogrellier/feature-selection-with-null-importances). Drawing inspiration from this method, we implemented the following steps for feature selection: (1) Eliminate features exhibiting zero variance; (2) Employ random forest (default parameters) to compute the score for each feature within the original dataset (recorded as the actual score); (3) Randomly shuffle labels and employ random forest (default parameters) to derive scores for each feature. The resulting scores after label shuffling denote the null score for each feature. This process is repeated 1000 times to obtain the score distribution, and recorded as null scores for subsequent use. The custom scoring function is defined as follows:

(2)$$finalscore=actualscore/nullscore{s}_{mean}$$

The final score for each feature is computed using formula (2), features with a final score of 0 are eliminated, and the top 5% of features are selected as the ultimate input variables.

The mRMR algorithm is predicated on the principle of maximizing the relevance of selected features with respect to the target variable, while concurrently minimizing the redundancy among these features. The dual objectives of the mRMR algorithm ensure that the selected feature subset has a high degree of predictive power for the target variable, and is devoid of superfluous or duplicative information that could impair model performance. The mRMR algorithm operates by iteratively evaluating the mutual information between each feature and the target variable, thereby quantifying the relevance of each feature. Simultaneously, it assesses the mutual information among the features themselves to gauge their redundancy. The algorithm seeks to construct a feature subset where the average mutual information between the features and the target is maximized, and the average mutual information among the features is minimized. This approach facilitates the selection of a feature set that is both highly informative and minimally redundant, thus striking a balance between relevance and redundancy [27]. To implement the mRMR algorithm, we utilized the mrmr_classif function available in the mrmr package (version: 0.2.8) for Python (https://github.com/smazzanti/mrmr). The parameter K was set to match the number of features selected by the Null importances method, while retaining default values for the remaining parameters.

Elastic net is a regularization method that integrates the strengths of both the Ridge (L2 norm) and Lasso (L1 norm) regularization techniques. This integration is achieved by combining their penalty terms into a single formulation, thereby enabling the method to acquire the benefits of both approaches: Lasso’s capability for feature selection through its tendency to produce coefficients that are exactly zero, and Ridge’s ability to handle multicollinearity by more evenly distributing the penalty across all coefficients [44]. The sklearn.linear_model.ElasticNet class (https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html) within the sklearn library was utilized to implement Elastic net for feature selection under the default parameter settings. These default parameters, including the mix ratio between L1 and L2 regularization, were retained to provide a balanced regularization approach suitable for a wide range of datasets, as well as to ensure reproducibility and comparability of the results.

SVM serves in both classification and regression tasks. It aims to identify a decision boundary, often referred to as a hyperplane, which effectively separates data points into distinct categories. This is accomplished by maximizing the margin, which is the separation distance between the boundary and the nearest datapoint known as the support vector. The efficacy of SVM is particularly evident in its handling of high-dimensional data and non-linear problem sets, where it shows robust generalization capabilities [30]. The sklearn.svm.SVC class (https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) within the sklearn library was used to implement SVM. Two parameters must be considered when training an SVM with the Radial Basis Function (RBF) kernel: C and gamma. The parameter C trades off misclassification of training examples against simplicity of the decision surface. A low C makes the decision surface smooth, while a high C aims to correctly classify all training examples. gamma defines how much influence a single training example has. The larger the gamma value, the closer the other examples must be to be affected. During parameter optimization, C and gamma are spaced exponentially apart in order to select suitable values (https://scikit-learn.org/stable/modules/svm.html#svm-classification). The optimization ranges for parameters C and gamma were set to [0.001, 0.01, 0.1, 1, 10, 100, 1000] and [1, 0.1, 0.01, 0.001, 0.0001], respectively.

DT performs classification by querying features within the data and posing a series of questions. Each question serves as a node within the tree structure and segments data items into different child nodes based on the answers, thus enabling the classification process. This hierarchical arrangement of queries establishes the tree structure characteristic of decision trees. Compared to other classifiers such as neural networks [46], decision trees offer greater interpretability by leveraging straightforward, data-based inquiries in an understandable manner. Nevertheless, minor alterations in the input data can occasionally result in substantial modifications within the constructed tree [31]. The sklearn.tree.DecisionTreeClassifier class (https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) within the sklearn library was utilized to implement DT. The parameter max_depth represents the maximum depth of the tree. Typically, max_depth = 3 is set as the initial tree depth to obtain a preliminary evaluation of how the tree fits the data, and then the depth is increased. Controlling the tree size through max_depth helps to prevent overfitting (https://scikit-learn.org/stable/modules/tree.html). The optimization ranges for the max_depth parameter are set from 3 to 10 in steps of 1.

RF comprises numerous autonomously trained decision trees. The ultimate prediction relies on aggregating the outcomes from all trees [47]. This approach is used extensively in prediction tasks, accommodating both large and small sample datasets alongside high-dimensional feature spaces. It offers the advantages of high accuracy and requiring minimal parameter tuning. Additionally, it can readily adapt to diverse ad-hoc learning tasks and provides valuable information on feature importance [32]. The sklearn.ensemble.RandomForestClassifier class (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) within the sklearn library was used to implement RF. The parameter n_estimators represents the number of trees in the forest and is the primary parameter for adjustment when utilizing RF, with a default value of 100. Of note, the results stop getting significantly better beyond a critical number of trees (https://scikit-learn.org/stable/modules/ensemble.html#forest). The optimization ranges for the parameter n_estimators are set from 50 to 150 in steps of 10.

XGBoost is a gradient boosting tree model utilized for addressing classification and regression challenges. It enhances model performance through the iterative construction of multiple decision trees, and integrates various optimization techniques to ensure efficient, scalable, and accurate prediction capabilities. XGBoost has demonstrated prowess in machine learning competitions and real-world applications. It excels in managing large-scale datasets and high-dimensional features, showcasing exceptional generalization capabilities [33]. We utilized XGBoost via the xgb.XGBClassifier class (https://xgboost.readthedocs.io/en/stable/python/python_api.html) within the Python-based XGBoost Package. The parameter learning_rate denotes the boosting learning rate. It is instrumental in controlling overfitting through step size shrinkage, which is set to 0.01 [23]. n_estimators, akin to RF, signifies the number of boosting rounds. The parameter gamma indicates the minimum loss reduction required to further partition a leaf node within the tree. A higher gamma value results in a more conservative algorithm. subsample represents the subsample ratio of training instances. When set above 0.5, XGBoost randomly samples more than half of the training data before tree growth, effectively mitigating overfitting (https://xgboost.readthedocs.io/en/stable/). The optimization ranges for parameter n_estimators are set from 50 to 150 in steps of 10, while for gamma and subsample the ranges are [0.2, 0.4, 0.6, 0.8, 1] and [0.6, 0.7, 0.8, 0.9, 1], respectively.