Support vector machines
An SVM draws the boundary that keeps the widest possible safety gap between two classes, and only the borderline examples decide where it goes.
- 7 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
A support vector machine draws the dividing line that leaves the widest possible safety gap between two groups.
Picture a school corridor during a fire drill. Two lines of students walk in opposite directions, and a teacher paints a stripe on the floor to separate them. A careless teacher paints it anywhere in the gap. A careful teacher paints it right down the middle, as far from both lines as possible, so a stumbling student does not cross into oncoming traffic.
A support vector machine (SVM) is the careful teacher. Among all the boundaries that separate two classes, it picks the one with the maximum breathing room.
Why it exists
Many different lines can separate the same two groups of examples. A line that grazes past several training examples is fragile — tomorrow's slightly different example lands on the wrong side.
The SVM's answer: make the margin — the empty corridor between the boundary and the nearest examples on each side — as wide as possible. A wide margin means small surprises do not flip the answer.
Here is the surprising part. Only the few examples standing at the edge of the corridor decide where the stripe goes. Those edge examples are called support vectors, because they alone hold the boundary in place. Every example deep inside its own group could be deleted and nothing would move.
How it works
not admitted margin admitted
x x │◄────────►│ o o
x x │ │ o
x x ─────┤ stripe ├───── o o o
x │ │ o
edge x o edge
(support (support
vector) vector)A real example you have seen
Before deep learning took over, the spam filter in your inbox was often an SVM. Handwriting readers that sorted postal codes, and face detectors in early digital cameras, leaned on SVMs too. On small tabular datasets — a few hundred rows of patient measurements, say — they remain a strong, hard-to-beat choice today.
Remember this
- An SVM picks the boundary with the widest margin, not any boundary that works.
- Only the borderline examples — the support vectors — decide where it sits.
- Wide margins buy tolerance: a slightly unusual new example still lands on the right side.
What to learn next
- The kernel trick — how this linear machine learns curved boundaries.
- Logistic regression — the probabilistic sibling, one loss function apart.
- Overfitting and underfitting — what the
Cknob is actually trading.
Developer — Code and libraries.
Setup
pip install scikit-learnOutputs verified with scikit-learn 1.7.2 on CPU.
Admissions by exam scores
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(0)
# entrance-exam scores (maths, english): 1 = admitted, 0 = not admitted
admitted = rng.normal([70, 75], 6, (20, 2))
rejected = rng.normal([45, 50], 6, (20, 2))
X = np.vstack([admitted, rejected])
y = np.array([1] * 20 + [0] * 20)
for C in (0.01, 1.0, 100.0):
m = SVC(kernel="linear", C=C).fit(X, y)
margin = 2 / np.linalg.norm(m.coef_[0])
print(f"C={C:>6}: support vectors={m.n_support_.sum():2d} margin width={margin:.1f} marks")
borderline = np.array([[57.0, 62.0]])
m = SVC(kernel="linear", C=1.0).fit(X, y)
print("decision value for a borderline student:", round(m.decision_function(borderline)[0], 2))C= 0.01: support vectors= 4 margin width=13.4 marks C= 1.0: support vectors= 3 margin width=11.3 marks C= 100.0: support vectors= 3 margin width=11.3 marks decision value for a borderline student: -0.65
The walkthrough
C is the strictness knob. It prices margin violations. Small C says "keep the corridor wide, forgive a few students inside it" — you get the 13.4-mark margin held by 4 support vectors. Large C says "violations are expensive", narrowing the corridor. Here the classes separate cleanly, so C=1 and C=100 land on the same 3 borderline students and stop changing. On messy, overlapping data, cranking C up is a straight road to overfitting.
Only 3 or 4 points matter out of 40. That is the SVM signature. m.support_ lists their indices; the other 36 students could vanish without moving the boundary a millimetre.
decision_function returns a signed distance from the stripe, in margin units. Our borderline student scores -0.65: on the "not admitted" side, well inside the corridor — a close call, and the number says so. predict only reports the sign and throws that nuance away.
margin = 2 / np.linalg.norm(m.coef_[0]) converts the fitted coefficients into the corridor's width. Watching this number shrink as C grows makes the trade-off concrete.
Common mistakes
Not scaling features. SVMs measure distances, so a feature with big units dominates the margin geometry. These exam scores share one scale; real tables do not. Wrap the model as make_pipeline(StandardScaler(), SVC(...)) by default.
Needing probabilities and reading decision_function as one. -0.65 is not "35% admitted". Pass SVC(probability=True) — it fits an extra calibration model, costing an internal cross-validation — or keep decision values and set a threshold directly.
Training a kernel SVM on huge data. SVC training cost grows roughly with the square of the sample count. Past ~50k rows it crawls. Use LinearSVC or SGDClassifier(loss="hinge") for large linear problems.
Tuning C on the test set. C is a hyperparameter like any other. Choose it with cross-validation; touch the test set once, at the end.
Try it yourself
Move one rejected student to [68, 72] — deep inside admitted territory. Refit with C=0.01 and C=100, printing support-vector counts and margins. Watch the strict model contort around the intruder while the forgiving one accepts a mistake and keeps the corridor wide.
What to learn next
- The kernel trick — how this linear machine learns curved boundaries.
- Logistic regression — the probabilistic sibling, one loss function apart.
- Overfitting and underfitting — what the
Cknob is actually trading.
Researcher — Mathematics and papers.
The optimisation problem
For training pairs $(x_i, y_i)$ with $y_i \in {-1, +1}$, the soft-margin SVM solves:
$$ \min_{w, b, \xi} \;\; \frac{1}{2} \lVert w \rVert^2 + C \sum_{i=1}^{n} \xi_i \quad \text{s.t.} \quad y_i (w^\top x_i + b) \geq 1 - \xi_i, \;\; \xi_i \geq 0 $$
Where:
- $w \in \mathbb{R}^d$ — the boundary's normal vector; $b$ — the offset.
- $\xi_i$ — slack variables measuring how far example $i$ violates the margin.
- $C > 0$ — the penalty per unit of violation.
- The geometric margin is $2 / \lVert w \rVert$, so minimising $\lVert w \rVert^2$ maximises the corridor.
Equivalently, in unconstrained form, the SVM minimises regularised hinge loss:
$$ \frac{1}{n}\sum_{i=1}^{n} \max(0,\, 1 - y_i(w^\top x_i + b)) + \lambda \lVert w \rVert^2 $$
with $\lambda = 1/(nC)$. The hinge is flat for confidently correct points — their gradients vanish, which is exactly why non-support-vectors do not matter.
The dual, and why it matters
The Lagrangian dual is:
$$ \max_{\alpha} \; \sum_i \alpha_i - \frac{1}{2} \sum_{i,j} \alpha_i \alpha_j y_i y_j \, x_i^\top x_j \quad \text{s.t.} \quad 0 \leq \alpha_i \leq C, \;\; \sum_i \alpha_i y_i = 0 $$
with $\alpha_i$ the dual variable for example $i$; the solution gives $w = \sum_i \alpha_i y_i x_i$. Support vectors are exactly the points with $\alpha_i > 0$. Crucially, data enters only through inner products $x_i^\top x_j$ — the door through which the kernel trick walks in.
Complexity
Kernel SVM solvers (SMO; Platt, 1998) cost between $O(n^2 d)$ and $O(n^3 d)$ in practice, with $O(n_{sv} d)$ per prediction ($n_{sv}$ = number of support vectors). Linear SVMs escape this: LIBLINEAR's coordinate descent and Pegasos-style SGD (Shalev-Shwartz et al., 2011) train in $O(nd)$ per epoch, serving text classification at millions of examples.
Papers and context
- Cortes and Vapnik (1995), Support-vector networks — the soft-margin SVM as used today.
- Boser, Guyon and Vapnik (1992) — the hard-margin, kernelised original.
- Margin theory: generalisation bounds depend on the margin distribution rather than dimension count (Vapnik, 1998; Bartlett and Shawe-Taylor, 1999), the theoretical excuse for the whole enterprise.
Multi-class is handled by one-vs-one ( SVC) or one-vs-rest (LinearSVC) reductions; Crammer and Singer (2001) give a single joint formulation. On tabular data, gradient-boosted trees have largely displaced SVMs; on small clean datasets and in metric-learning pipelines, margins live on.
What to learn next
- The kernel trick — how this linear machine learns curved boundaries.
- Logistic regression — the probabilistic sibling, one loss function apart.
- Overfitting and underfitting — what the
Cknob is actually trading.