BayesNet/docs/BoostAODE.md

BoostAODE Algorithm Operation

The algorithm is based on the AdaBoost algorithm with some new proposals that can be activated using the following hyperparameters.

Hyperparameters

The hyperparameters defined in the algorithm are:

• repeatSparent (boolean): Allows dataset variables to be repeated as parents of an SPODE. Default value: false.

• maxModels (int): Maximum number of models (SPODEs) to build. This hyperparameter is only taken into account if repeatSparent is set to true. Default value: 0.

• order ({"asc", "desc", "rand"}): Sets the order (ascending/descending/random) in which dataset variables will be processed to choose the parents of the SPODEs. Default value: "desc".

• convergence (boolean): Sets whether the convergence of the result will be used as a termination condition. If this hyperparameter is set to true, the training dataset passed to the model is divided into two sets, one serving as training data and the other as a test set (so the original test partition will become a validation partition in this case). The partition is made by taking the first partition generated by a process of generating a 5 fold partition with stratification using a predetermined seed. The exit condition used in this convergence is that the difference between the accuracy obtained by the current model and that obtained by the previous model is greater than 1e-4; otherwise, one will be added to the number of models that worsen the result (see next hyperparameter). Default value: false.

• tolerance (int): Sets the maximum number of models that can worsen the result without constituting a termination condition. Default value: 0.

• select_features ({"IWSS", "FCBF", "CFS", ""}): Selects the variable selection method to be used to build initial models for the ensemble that will be included without considering any of the other exit conditions. These models also do not update or use the weights used by the Boosting algorithm, and their significance is set to 1.

• threshold (double): Sets the necessary value for the IWSS and FCBF algorithms to function. Accepted values are:

  • IWSS: threshold \in [0, 0.5]
  • FCBF: threshold \in [10^{-7}, 1]

  Default value is -1 so every time any of those algorithms are called, the threshold has to be set to the desired value.

• predict_voting (boolean): Sets whether the algorithm will use model voting to predict the result. If set to false, the weighted average of the probabilities of each model's prediction will be used. Default value: true.

• predict_single (boolean): Sets whether the algorithm will use single-model prediction in the learning process. If set to false, all models trained up to that point will be used to calculate the prediction necessary to update the weights in the learning process. Default value: true.

Operation

The algorithm performs the following steps:

1. Initialization

   • If select_features is set, as many SPODEs are created as variables selected by the corresponding feature selection algorithm, and these variables are marked as used.

   • Initial weights of the examples are set to 1/m.

2. Main Training Loop:

   • Variables are sorted by mutual information order with the class variable and processed in ascending, descending or random order, according to the value of the order hyperparameter. If it is random, the variables are shuffled.

   • If the parent repetition is not established, the variable is marked as used.

   • A SPODE is created using the selected variable as the parent.

   • The model is trained, and the class variable corresponding to the training dataset is calculated. The calculation can be done using the last trained model or the set of models trained up to that point, according to the value of the predict_single hyperparameter.

   • The weights associated with the examples are updated using this expression:

     • w_i · e^α_t (if the example has been misclassified)

     • w_i · e^-α_t (if the example has been correctly classified)

   • The model significance is set as α_t.

   • If the convergence hyperparameter is set, the accuracy value on the test dataset that we separated in an initial step is calculated.

3. Exit Conditions:

   • ε_t > 0.5 => misclassified examples are penalized.

   • Number of models with worse accuracy greater than tolerance and convergence established.

   • There are no more variables to create models, and repeatSparent is not set.

   • Number of models > maxModels if repeatSparent is set.