SVM (support vector machine)

A support vector machine, or SVM, draws the boundary between classes that keeps the widest possible gap to the nearest examples on each side.

Think of laying a footpath between two villages' fields. Many paths would separate the fields, but the sensible one runs down the middle, as far from both fences as possible — small surveying errors then never put the path inside anyone's field. The SVM chooses its decision boundary the same way: not any separator, but the maximum-margin one. The few examples pressing right up against the margin are the support vectors — they alone define the boundary, and every other point could be deleted without changing it.

The second idea is the kernel trick. Real classes are rarely separable by a straight line. A kernel implicitly lifts the data into a higher-dimensional space where a straight boundary does exist, without ever computing the lift. The practical menu is short: linear kernel for text and wide data, RBF (radial basis function) kernel for general curved problems.

SVMs were the dominant classifier of the 2000s and remain excellent for small-to-medium datasets — thousands to tens of thousands of rows — especially with many features and few examples. They need feature-scaling, and training scales poorly to millions of rows, which is the terrain where tree ensembles and neural networks took over.

Where to go next#

• Full lesson: Classification
• Related terms: classification, feature-scaling, logistic-regression, knn