Extended definition
Gradient boosting is an ensemble technique that builds a strong model by summing many weak models, usually shallow decision trees, trained in sequence. The idea, formalized by Friedman (2001), is to treat learning as a gradient descent in function space: each new tree is fit to correct the errors of the ensemble accumulated so far, approximating the negative gradient of the loss function. The final model is a weighted sum of these trees, and a small learning rate controls how much each one contributes. Unlike random forest, which trains trees in parallel and combines them by averaging, boosting is strictly sequential: each tree depends on the previous ones. Natekin and Knoll (2013) describe the method’s flexibility, which adapts to different loss functions and therefore to regression, classification, and ranking. Scalable implementations, chiefly the XGBoost of Chen and Guestrin (2016) and LightGBM, made gradient boosting the de facto standard on tabular data.
When it applies
Gradient boosting applies as a first choice on structured tabular data, where it often outperforms neural networks and linear models. It applies well to mixtures of numerical and categorical variables, captures interactions and nonlinearities without extensive manual engineering, and handles complex relationships between predictors and target. It is strong in competitions and in applied problems of risk prediction, customer classification, fraud detection, and demand forecasting. It applies when high predictive performance is wanted and there is time to tune hyperparameters. Modern implementations offer native handling of missing values, regularization, and efficient parallelization, which makes use on large datasets feasible. Combined with SHAP, it also offers interpretability at the level of each variable’s contribution.
When it does not apply
Gradient boosting does not apply without care for overfitting: being sequential and greedy, it fits noise if the number of trees, depth, and learning rate are not regularized and validated. It does not apply well to very high-dimensional, sparse data such as raw text or images, where specialized models dominate. It is not the best option when simple, direct interpretability is mandatory: the ensemble of hundreds of trees is not readable like a regression. It does not apply when the tuning budget is minimal; the method is sensitive to hyperparameters, and a poorly tuned model loses to more robust alternatives such as random forest. And it does not apply to extrapolation beyond the training range: trees do not project trends outside the observed domain.
Applications by field
- Risk and finance: default models, credit scoring, and fraud detection over tabular data.
- Health and epidemiology: prediction of clinical outcome from structured variables, with interpretation by variable contribution.
- Marketing and operations: churn, demand, and propensity forecasting, where predictive performance is the central criterion.
- Applied social sciences: predictive modeling on survey and administrative data, as a complement to classical inference.
Common pitfalls
The first pitfall is letting the model grow without regularization: many deep trees with a high learning rate lead to silent overfitting. The second is failing to separate validation from test honestly; intensive hyperparameter tuning leaks information if evaluation is not careful. The third is confusing raw variable importance with causality: a predictor’s importance in boosting is association, not causal effect. The fourth is applying the method to raw text or images, where it loses to purpose-built architectures. The fifth is forgetting that trees do not extrapolate: using a boosting model to predict outside the training range produces flat, misleading estimates.