The Brain Is a World Model

Why do we see things that are not physically there?

Visual illusions are often described as mistakes made by the brain. But I believe they reveal something much more interesting.

The brain is a world model.

It does not simply record the light entering our eyes, frame by frame, like a camera. Instead, through years of experience, it learns the statistical structure of the world: how objects move, what colors they tend to have, what usually happens next, and which visual features tend to belong together.

What we perceive is generated using this learned model.

And prediction may be one of the main algorithms by which the brain builds it.

Perception is more than receiving information

The visual information reaching the brain is always incomplete.

Neural processing takes time. Objects move. We move. Our eyes move several times every second. Parts of the world are hidden behind other objects. Lighting changes constantly.

If the brain merely waited for sensory information and reacted to it afterward, perception would always lag behind the world.

A more effective strategy is to learn how the world behaves and continuously predict what is likely to happen next.

In predictive processing, the brain does something like this:

past experience → internal model → prediction → sensory input → prediction error → model update

Prediction is therefore not simply a way to guess the future. Over a lifetime, prediction becomes a way to learn the structure of the world itself.

This idea has been central to our research for many years.

An illusion of motion from a model that learned the world

In 2018, we asked whether a neural network trained only to predict natural visual sequences could reproduce a human visual illusion.

We used a predictive neural network called PredNet and trained it on natural first-person videos. The network was not taught about visual illusions. It was not told how objects should move. Its task was simply to predict the next image from previous images.

Through this task, it had to discover regularities in the visual world.

We then showed the trained network the Rotating Snakes illusion—a completely static image that nevertheless produces a vivid sensation of rotation in human observers.

Remarkably, the network predicted motion in the illusion in a direction consistent with human perception.

Why?

Our interpretation was that the illusion was not an arbitrary failure of the system. It was a consequence of what the system had learned about normal visual dynamics.

The same internal model that is useful for predicting motion in the real world can produce an incorrect prediction when it encounters an unusual artificial pattern.

In this view, an illusion is not simply a bug.

It is a fingerprint of the world model inside the brain.

From motion to color

More recently, we asked whether the same principle could explain an even stranger phenomenon: seeing colors in a black-and-white image.

Benham's top is a disk containing only black and white patterns. When the disk rotates, many observers perceive faint colors—often reds, greens, blues, or yellows—even though there is no physical color in the stimulus.

This phenomenon has been known for nearly two centuries.

In our 2026 study, we trained predictive neural networks on videos and then presented them with achromatic patterns related to Benham's top.

The networks generated artificial subjective colors.

But the most revealing result came when we changed the training environment.

When we systematically changed the colors of moving objects in the training videos, the colors later generated by the network also changed.

In other words, the illusion carried a trace of the visual statistics that the network had experienced during learning.

This suggests a simple possibility.

During everyday life, the visual system repeatedly experiences relationships such as:

this kind of motion + this kind of visual structure + this kind of color

Through predictive learning, these properties become statistically associated.

Later, when an unusual black-and-white stimulus produces a similar motion pattern, the visual system may infer a color that is statistically plausible—even though that color is absent from the physical stimulus.

The perceived color would then come not from the stimulus alone, but from the brain's learned model of the world.

The world itself contains color biases

An interesting clue comes from work by Bevil Conway and colleagues.

They analyzed more than 20,000 photographs containing salient objects and compared the colors belonging to objects with those belonging to their backgrounds.

The distributions were not the same.

Objects tended, statistically, to be warmer in color and more saturated than backgrounds. Color information alone even contained useful information for distinguishing some broad categories of objects.

This is important because it tells us that object color is not statistically independent of the structure of natural scenes.

The real world contains regularities.

And a brain that spends its life learning the world has the opportunity to learn those regularities.

Even a “static” object moves across the retina

There is another step that I think is particularly interesting.

We normally divide visual scenes into “moving objects” and “static objects.”

But from the point of view of the retina, the distinction is less simple.

Imagine walking through a forest.

The trees are not physically moving. Yet their images move across your retina because you are moving.

Nearby objects sweep rapidly across the visual field, while distant backgrounds move more slowly. This phenomenon—motion parallax—is one of the fundamental sources of information about depth.

So an object does not need to move by itself to generate an object-specific motion signal.

Observer motion transforms spatial structure into temporal structure.

From the perspective of a learning visual system, color, object identity, depth, and motion are therefore not separate problems. They are repeatedly experienced together.

A red fruit on a green background is not merely a particular spatial arrangement of colors. As we approach it, turn our head, or move our eyes, the fruit and background generate characteristic temporal relationships as well.

Over millions of visual experiences, the brain can learn these joint statistics.

This leads to a broader hypothesis:

The visual system learns not only what the world looks like, but how the world changes.

And because change depends both on objects moving and on ourselves moving, the distinction between spatial statistics and motion statistics may be much weaker for the brain than it appears to us conceptually.

The brain does not reconstruct the world. It models it.

It is tempting to imagine vision as a pipeline:

world → eyes → brain → perception

But I think the relationship is better described as a loop:

world → sensory input → internal model → prediction → comparison with the world → learning

The brain continuously updates a model of its environment and uses that model to interpret new sensory signals.

Most of the time, this process works extraordinarily well. We perceive stable objects, colors, depth, and motion despite noisy and incomplete sensory information.

Occasionally, however, an artificial stimulus falls into a statistical gap between the real world and the model learned from it.

Then the model reveals itself.

We see a static image move.

We see color in black and white.

These experiences may seem like failures of perception. But they may actually expose the computational principles that make normal perception possible.

Illusions are windows into the learned world model

This is why I have long been interested in visual illusions.

An illusion is a special experiment designed—sometimes intentionally, sometimes accidentally—to separate physical reality from perceptual reality.

When those two realities diverge, we can ask:

Why did the brain choose this particular interpretation?

The answer may often be found not in the stimulus itself, but in the statistics of the world in which the brain developed.

Our 2018 study suggested that a predictive network that learns natural visual dynamics can acquire human-like motion illusions.

Our 2026 study suggests that predictive learning can also bind motion and color statistics strongly enough to generate colors that are physically absent.

And studies of natural scenes show that objects and backgrounds really do have different color statistics.

Together, these observations point toward a common idea:

The brain is a world model, built through prediction.

Perception is not simply a readout of the present.

It is the present interpreted through everything the brain has learned about the world.

And sometimes, when we see something that is not there, we may actually be seeing something deeper:

the structure of the world that has been learned inside us.

Search This Blog

Eiji Watanabe Lab @ NIBB, Japan