When Do Neural Nets Outperform Boosted Trees on Tabular Data?
Tabular data is one of the most commonly used types of data in machine learning. Despite recent advances in neural nets (NNs) for tabular data, there is still an active discussion on whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data.
Further complicating this debate is the recent emergence of TabPFN, a prior-data fitted network, which learns tabular data in fundamentally different ways, compared to its predecessors. TabPFN does not fit its parameters to training data; at inference time, it learns patterns in-context from the training data and uses those patterns to generate predictions on a test set.
I was fortunate to have the opportunity to contribute to a new paper which will be appearing at NeurIPS 2023, Datasets and Benchmarks Track. We conducted the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and found few consistent differences in performance between NNs and GBDTs.
By contrast, TabPFN, despite being limited to training sets of 3000 samples or fewer, outperformed all other algorithms on average (this latter result was my main contribution to the work).
This is a very surprising outcome, and no doubt there will be future works investigating how TabPFN is able to achieve this feat. In the meantime, check out the full paper on arXiv, as it contains many important details and insights beyond what I am able to cover in a brief blog post.