Ensemble Processes, what are they like? Ensemble methods is a machine learning technique that combines many basic models to create one optimal predictive model. To better understand this concept, let us take a step back to the ultimate goal of machine learning and model building. This will make more sense when I delve into concrete instances and why Ensemble approaches are used.
To a large extent, I will use Decision Trees to outline the concept and practicality of Ensemble Methods (however it is important to note that Ensemble Methods do not only pertain to Decision Trees).
The Decision Tree shall determine the predictive value on the basis of a set of questions and conditions. For example, this simple Decision Tree decides whether or not a person should play outside. The tree takes into account a variety of weather variables and, provided each factor, either makes a decision or asks another question. In this example, every time it’s overcast, we’re going to play outside. But if it’s raining, do we have to wonder if it’s windy or not? If it’s windy, we won’t play. But because there was no breeze, tie those shoelaces close because they were going outside to play.
Decision Trees can also solve quantitative problems in the same style. In the Tree to the left, we want to know whether or not to invest in commercial real estate. Is it a building for the office? A factory, huh? The building of the apartment? Healthy economic conditions, huh? Weak economic conditions, huh? How much is the investment going to return? These questions are answered and resolved by using this decision tree.
There are several considerations that need to be taken into account when making Decision Trees: on what characteristics do we make our decisions? What is the threshold for classifying each question as a yes or no answer? In the first decision tree, what if we were to ask ourselves whether or not we had friends to play with. If we have friends, we’re going to play every time. If not, we might still be asking ourselves questions about the weather. By adding an additional query, we hope to describe the Yes and No groups more clearly.
That’s where the Ensemble Approaches come in handy! Rather than depending on one Decision Tree and assuming that we made the right decision on each split, Ensemble Methods enables us to take into account a sample of Decision Trees, measure the features to be used or questions to be asked on each split, and make a final predictor based on the aggregated results of the sampled Decision Trees.
Ensemble Process Forms
Bagging, or Aggregating Bootstrap. Bagging gets its name because it incorporates Bootstrapping and Aggregation to form a single ensemble model. In view of the data set, several bootstrapped subsamples are drawn. On each of the bootstrapped subsamples, a Decision Tree is constructed. After and subsample Decision Tree has been developed, an algorithm is used to aggregate over the Decision Trees to form the most efficient predictor. The picture below will help to explain:
Models for the Random Forest. Random Forest Models, with a slight tweak, can be thought of as Bagging. When deciding where to divide and how to make decisions, Bagged Decision Trees shall have the full range of features to choose from. Therefore, although the bootstrapped samples may be slightly different, the data will largely break off with the same features throughout each model. In contrast, Random Forest models decide where to divide on the basis of a random selection of features. Rather than splitting at similar features at each node throughout, Random Forest models set a differentiation level because each tree will be divided on the basis of different features. This level of differentiation provides a larger set for aggregate over, ergo producing a more accurate predictor. Please refer to the image for a better understanding.
Similar to Bagging, bootstrapped subsamples are drawn from a larger dataset. For each subsample, a decision tree is formed. HOWEVER, the decision tree is divided into different features (in this diagram the features are represented by shapes).
Summary
The goal of any machine learning problem is to find a single model that best predicts our desired outcome. Instead of making a model and hoping that it will be the best/most accurate predictor we can make, set methods take a myriad of models into account, and average those models to produce one final model. It is important to note that Decision Trees are not the only form of Ensemble Methods, but the most popular and relevant in Data Science today.
