A neural network, in the simplest mathematical sense, is a function built by stacking layers of a basic computation: take some inputs, multiply each by a weight, add a bias, push the result through a fixed activation function (a nonlinear shape like a sigmoid or ReLU), and pass the result on. Train the weights and biases on data and the network can learn to recognise faces, translate languages, drive cars, and beat world champions at strategy games. The big practical surprise of the 21st century has been how much such simple objects can do.
The mathematical justification for why networks can in principle do anything dates back to 1989, when George Cybenko proved a result now known as the Universal Approximation Theorem: a feedforward neural network with a single hidden layer, given enough neurons, can approximate any continuous function on a closed bounded subset of to arbitrary accuracy. Kurt Hornik strengthened the result in 1991 by showing it holds for essentially any non-polynomial activation function. The theorem is the mathematical licence for the entire field of machine learning: it guarantees that the model has, in principle, enough representational power to fit whatever pattern the data contains.
This article is about what the theorem says, what it really means (and what it does not mean), and why it has remained a foundational result even as the field has moved on to architectures Cybenko could never have imagined.
What the theorem says
Cybenko’s statement, in essentially modern language: let be a continuous sigmoidal function — meaning as and as . Then the set of all functions of the form
— finite linear combinations of shifted, scaled, rotated copies of — is dense in the space of continuous functions on any compact subset of . In plain words: for any continuous target function on a closed bounded region and any desired accuracy , there exist some number of “neurons” and some choice of weights , biases , and output coefficients such that the network’s output differs from by at most everywhere in the region.
The structure on the right — sum of activation functions applied to affine transformations of the input — is exactly the formula for a neural network with one hidden layer of neurons. The theorem says that this very simple architecture is, in principle, universal: nothing about the structure of the network prevents it from representing any continuous behaviour you might want.
A picture of how it works
The cleanest intuition comes from the one-dimensional case. Imagine you want to approximate a smooth function on an interval. With sigmoid-shaped activations, the difference of two sigmoids creates a localised “bump” of width controlled by their relative shift. By taking many such bumps and adjusting their amplitudes, you can chase any desired height at any desired location, much like building a curve out of tiny rectangular blocks.
The figure below shows a target function (a smooth sine-modulated wave) and a step-function approximation built from twelve constant-height bumps. The approximation is rough — a network with twelve neurons cannot do better than rough — but the principle is clear: increasing the number of bumps would tighten the fit. With enough neurons, the green staircase approaches the black curve.
In higher dimensions the same logic applies, only the bumps live on a multi-dimensional grid. The universal approximation theorem is, at heart, this elementary statement made precise and rigorous: shifted, scaled copies of a single nonlinear shape can build any continuous landscape.
What the proof really uses
Cybenko’s original proof was an application of the Hahn–Banach theorem in functional analysis. The argument is short but slick: assume the set of expressible functions is not dense, and derive a contradiction with the properties of sigmoidal functions. The result is that any continuous target can be approximated to within any positive tolerance — but the proof itself is non-constructive. It guarantees that good weights exist; it does not tell you how to find them.
Hornik’s 1991 extension replaced the specific sigmoidal assumption with a much more general one: as long as the activation function is continuous and non-polynomial, the same universality holds. The reason is essentially that polynomials of bounded degree form a finite-dimensional subspace and cannot be dense in the infinite-dimensional space of continuous functions, whereas any non-polynomial activation function generates enough flexibility under finite combinations to be dense. This explains why almost every activation that has ever been used in practice — sigmoid, tanh, ReLU, leaky ReLU, GELU, swish — works: they are all non-polynomial.
What the theorem does not say
It is worth being precise about what universal approximation does not deliver, because confusion about this has been the source of much overclaiming in popular descriptions of AI.
No bound on the number of neurons. The theorem says some number of neurons suffices for any given accuracy and any given target function . The number may be astronomical. For highly wiggly target functions, the required width of a single-layer network can grow exponentially with the input dimension — the so-called curse of dimensionality. The theorem promises representational adequacy but says nothing about whether the right network is of practical size.
No guarantee of trainability. Even if a perfect network of moderate size exists, the learning algorithm — usually some variant of gradient descent applied to a training loss — has to actually find the right weights. Optimisation in high dimensions can fail in many ways: local minima, saddle points, vanishing or exploding gradients, ill-conditioning. The theorem does not say that gradient descent will converge to a good network; it only says that a good network exists somewhere in the parameter space.
No guarantee of generalisation. A network that fits the training data perfectly may behave terribly on unseen inputs. Generalisation depends on the relationship between the network’s expressive capacity, the amount of training data, the structure of the data distribution, and various statistical-learning quantities like Rademacher complexity and VC dimension. None of this is touched by universal approximation.
The theorem is therefore necessary but not sufficient for the success of neural networks. It guarantees that the model class is expressive enough to contain whatever function the data is describing. It does not guarantee that the data, the loss, and the optimiser are jointly able to find that function.
Width versus depth
The theorem, as Cybenko proved it, uses a single hidden layer of arbitrary width. The mathematical content is silent on whether deeper, narrower networks would be better. In practice, deep networks — networks with many hidden layers — dominate modern machine learning. The reason is not that depth is necessary in principle (universal approximation says it is not) but that depth is often more efficient.
Several theoretical results from the late 1980s onward — by Johan Håstad, Matus Telgarsky, and others — show that certain classes of functions require exponentially more neurons to represent with a shallow network than with a deep one. The compositional structure of deep networks — each layer building features out of the previous layer’s features — turns out to align well with the compositional structure of real-world data, in which images consist of objects that consist of parts that consist of textures. Universal approximation underwrites the entire field, but the practical successes of deep learning came from architectural choices that exploit composition rather than relying on raw width alone.
A foundational guarantee
The universal approximation theorem is the kind of result that enables a field rather than driving it. It tells researchers that the basic model — a feedforward network with non-linearities — is not fundamentally limited in what it can represent. It does not solve any actual problem; it just opens the door. Everything else — training algorithms, architectures, regularisation, data engineering — is built on top of that opened door.
Without the theorem, every advance in machine learning would carry a nagging worry: “Maybe this whole class of models simply cannot represent the function we want.” With the theorem, that worry is permanently retired. Cybenko’s 1989 paper, four pages of clean functional analysis, is the mathematical bedrock on which the modern AI revolution rests. The fact that the revolution then took another twenty-five years to arrive in earnest is itself evidence that representational adequacy is only the first step — but it is the step without which none of what followed would have been possible.
Frequently asked
Who proved the universal approximation theorem?
The first widely-cited version is due to George Cybenko, who proved in 1989 that a single hidden layer of sigmoid neurons suffices to approximate any continuous function on a compact subset of R^n. Kurt Hornik gave an independent and more general proof in 1991 showing the result holds for essentially any non-polynomial activation function. Later refinements by Allan Pinkus, Moshe Leshno, and others sharpened the result and extended it to various function classes.
Does the theorem mean neural networks can solve any problem?
No — and this is a common misunderstanding. The theorem promises that a sufficiently large network can represent any continuous function arbitrarily well. It says nothing about whether you can find the right weights by training, how much data you need, how long training takes, or whether the network will generalise to unseen inputs. It is a statement about expressive capacity, not about learnability. Many functions that exist in principle are completely out of reach of practical training algorithms.
If one hidden layer is enough, why do deep networks work better?
Depth changes the cost dramatically. The universal approximation theorem says a single layer suffices, but the required number of neurons may be astronomically large for hard functions. Deep networks can represent the same functions exponentially more efficiently in some cases — a result formalised by Hastad in the 1980s for Boolean circuits and extended to neural networks by Telgarsky and others. In practice, deep networks also seem to be easier to train and to generalise better, though the theoretical reasons are still actively researched.
What activation functions work?
Almost any reasonable one. The original Cybenko proof used sigmoid-shaped functions. Hornik's extension covers any continuous non-polynomial activation, including ReLU (max(0, x)), tanh, leaky ReLU, GELU, and many others. The mathematical reason is that polynomials of bounded degree cannot approximate non-polynomial functions, but any non-polynomial activation introduces 'enough' nonlinearity that finite combinations of shifted, scaled copies span a dense subset of the continuous functions. ReLU networks in particular have been popular in modern deep learning because they train faster despite their simplicity.