An Introduction to Transfer Entropy
How information flows between systems
Description
Two things move together. A neuron fires, and a moment later another one fires too. The price of one asset climbs, and a second asset follows. The heart rate rises, then breathing quickens. We watch these paired signals and we want to say something we can almost never justify: that one of them is doing something to the other. Correlation tells us they dance in step. It stays completely silent on who is leading. For most of the twentieth century, that silence was simply the price of doing quantitative science on systems too tangled to take apart by hand.
Transfer entropy is the tool this book builds to break that silence. Introduced by the physicist Thomas Schreiber in a 2000 paper and developed since into a full method, it borrows the machinery of information theory — the same mathematics Claude Shannon assembled in the late 1940s to describe telephone lines — and repurposes it to ask a directed question. Not how much do these two signals share, but how much does knowing the past of one help us predict the future of the other, beyond what that other's own past already tells us. The word for what it measures is influence, with a direction attached.
The book is an entry point, written for people who have complex time series on their screens and a suspicion that something in the data is pushing something else. It sits between the equations and the lab, taking the concepts seriously without pretending the messy realities of finite, noisy measurement can be waved away. What follows is that path — from a familiar frustration to a working measure of directed information flow.
The question we’re asking : When two systems evolve together, how can we tell which one is actually influencing the other?What we’ll see : How a measure born from information theory learns to read direction in the flow between coupled systems, and what that buys the people who use it.
Table of contents
01Chapter 1 — The question correlation could never answer
Start with the frustration, because that is where the whole enterprise begins. Suppose we record two systems over time — call them source and target, though at the outset we have no right to those labels. We can compute how strongly they rise and fall together. A high correlation feels like a discovery, but it collapses the instant we press it. Maybe the source drives the target. Maybe the target drives the source. Maybe a third thing we never measured drives both, and the two we watched are just its shadows. Correlation is symmetric: swap the two signals and the number does not change. Direction is exactly the thing it throws away.
The older attempts to recover direction leaned on the intuition that causes come before effects. If knowing the past of one signal improves our prediction of another's future, perhaps the first is influencing the second. This is the logic behind Granger causality, the framework the economist Clive Granger formalized in the 1960s and was later recognized for. It was a real advance. But it was built on linear models — it assumes influence shows up as a straightforward, proportional relationship between past and future.
02Chapter 2 — Shannon's ghost, and the arrow he left out
Claude Shannon was not thinking about neurons or markets when he published his theory of communication in 1948. He was thinking about how to send messages down a wire without garbling them, and he needed a way to measure how much a message could carry. His answer was entropy: a number that captures how uncertain we are about what comes next. A signal that is perfectly predictable carries no information — we already knew what it would say. A signal full of genuine surprise carries a lot. Entropy puts a figure, in bits, on that surprise.
From this single idea the whole toolkit unfolds, and the book walks it out patiently. Mutual information asks how much learning one variable reduces our uncertainty about another — how many bits they share. It is a real improvement on correlation, because it catches nonlinear relationships that a straight line would miss. But it inherits correlation's original sin: it is symmetric. The bits shared between two signals are the bits shared, full stop. Swap them and nothing moves. Mutual information tells us the two systems know something about each other. It still cannot say who told whom.
03Chapter 3 — From equations to noisy real-world data
The definition is clean. Using it is not, and the book earns its keep by refusing to hide the difference. Transfer entropy is written in terms of probability distributions over the past and future states of two systems. In a textbook those distributions are simply given. In a lab you have a finite string of measurements — a few thousand samples, maybe far fewer — and you have to estimate the distributions from that. Every estimate carries error, and error in an information measure has a nasty habit of masquerading as signal.
Several practical decisions have to be made before a single number comes out, and each one bites. How far into the past should we look — one step, three, ten? Reach too short and you miss the influence; reach too far and you drown in parameters the data cannot support. How do we turn continuous measurements into something we can count probabilities over — bin the values into discrete boxes, or use a smarter estimator built for continuous data? The choice changes the answer. The book is candid that there is no universal recipe; there are trade-offs, and the honest analyst reports them.
04Chapter 4 — A measure that made asymmetry visible
Step back from the mechanics and transfer entropy turns out to be a small revolution in what we let ourselves ask of data. For most of quantitative history, the workhorse question was one of togetherness: do these things move as one? It is a fundamentally symmetric question, and it produced a fundamentally symmetric picture of the world — webs of association in which everything is related to everything and nothing is doing anything to anything. Transfer entropy replaces togetherness with direction. It lets us draw arrows, not just lines, between the parts of a system.
That reframing is why the same measure keeps surfacing in fields that share almost no vocabulary. In neuroscience, the question is which regions of the brain drive which others as a thought or a movement unfolds — an arrow through a network, not a static map of who lights up with whom. In finance and economics, it is which markets lead and which follow, which shocks propagate and in which direction. In physics and climate science, it is how influence moves through coupled dynamical systems. Different data, one question underneath: who is driving whom.
05Conclusion
We began with two signals moving in step and a question we had no clean way to settle: which one is leading. Transfer entropy answers by refusing the symmetric framing entirely. It takes Shannon's currency of bits, adds the one thing Shannon's own theory left implicit — an arrow — and hands back not a single number for the pair but two, one for each direction, whose difference is the influence we were after. What looked like an unanswerable question turns out to have been a question asked with the wrong tool.