Skip to content

The way the fish swim depends on their sensed surroundings

← Field Notes

Noise, and the observer who moves

·Jul 19, 2026
predictive codingactive inferenceMontyThousand Brains Theorycomputational psychiatrynotes

Our senses deliver a noisy signal, but the brain still has to commit to one account of what is out there, and in my opinion that commitment is the main problem that cognition is solving.

Under noise, the sensible thing to do is weight each piece of evidence by how reliable it is. When Marc Ernst and Martin Banks added visual noise to a size judgment, people shifted their weight toward touch (Ernst and Banks 2002). A statistician would call that weight precision (the inverse of the variance, or how tightly a signal sits around its true value), and an engineer would call it signal-to-noise.

Predictive coding treats the brain as something that predicts its own input and reacts mainly to what it got wrong (the prediction error). In this framework, precision sets how hard a prediction error pushes the estimate, which means that when precision on the senses is lowered, the brain’s priors (what it already expected before the input arrived) carry more of the weight. Harriet Feldman and Karl Friston argue that attention is mostly the brain turning this precision knob (Feldman and Friston 2010). However, the idea is hard to falsify, so I treat it as one view and not as settled fact.

This is also where the idea reaches psychiatry, because a system can mis-set its own precision even when the input it receives is clean. I want to be clear that these are hypotheses about mechanism, and they say nothing about anyone’s worth or character. Rick Adams and colleagues describe psychosis as aberrant precision, where sensory errors are weighted too heavily or priors too little (Adams et al. 2013). Autism has parallel accounts that disagree with each other on the direction of the change (Pellicano and Burr 2012), and behavior alone cannot separate these accounts (Sterzer et al. 2018).

A still observer can weight its cues and lean on its priors, but it cannot go and get more information, and to do that it has to move. Active inference treats action as a kind of inference, where the system both updates its beliefs to match the world and acts so that its input matches its predictions (Friston 2010). The figure below shows this loop as a search for car keys in a kitchen.

A first-person kitchen search for car keys. A gaze reticle checks the hooks, a left drawer, a right drawer, and a dish. A thought bubble holds a top-down map, and before each look a red question mark marks the model's prediction of where the keys are. Empty spots are ruled out and the guess narrows until the keys turn up in the dish, then the loop repeats.

Searching a kitchen for keys, with predictions on an internal mapA schematic first-person kitchen with window, cabinets, counter, two drawers, a mug, a bowl, a dish, and a hook rail. A teal gaze reticle moves between spots. A purple thought-bubble map shows shaded circles for how likely the keys are at each spot, and a red question mark for the current prediction. Ruled-out spots grey and the belief concentrates until a gold key appears in the dish. where are the keys? ? hooks left drawer right drawer dish
sense interpret act
Looking
the model bets on the hooks, go look
The sense, interpret, and act loop, shown as a search for car keys in a kitchen. Before each look, the internal model makes a prediction (the red '?') of where the keys are, and checking an empty spot rules that spot out, so the belief narrows until it lands on the dish. Teal shows looking and moving, and purple shows the mental map.

Monty is where I test this. Monty is an open-source sensorimotor learning system from the Thousand Brains Project, and it implements the idea that the neocortex learns objects by moving a sensor over them (Clay, Leadholm and Hawkins 2024). To recognize an object, Monty keeps an “evidence” score for each guess and updates it every time the sensor moves. That score is a heuristic, which means that it never becomes a probability. On clearly different objects Monty takes around 28 steps to recognize one, and with noise on similar objects it takes closer to 170.

My honors thesis makes one change to this. Monty’s matching tolerance (how close a sensed feature has to be to a stored one to count as a match) is a fixed number that someone has to choose by hand. I treat that tolerance as precision and let the system estimate it from experience instead. In practical terms, this turns the stored model into a real probability distribution, so that belief, precision, and free energy become numbers that I can check and break.

Once that is in place, I can miscalibrate precision on purpose. If I push the system to trust its predictions too much, the loop stops learning, because an empty spot no longer rules anything out and the guess keeps returning to where it expected the keys to be.

To be honest about the limits, Monty is an object recognition system and not a model of hallucination or delusion, and I do not have a clean result yet. However, I think it has real potential as a testbed for psychiatric conditions, and bridging that gap is the work I want to do. Computational psychiatry has been proposed as a bridge from neuroscience to the clinic (Huys, Maia and Frank 2016), and in epilepsy there are already whole-brain models individualized to single patients (Jirsa et al. 2017). In my future research, I hope to use embodied models like Monty to find candidate mechanisms and biomarkers for psychiatric disorders, with the long-term goal of precision psychiatry at the level of the individual patient.

References

  • Green, D. M., & Swets, J. A. (1966). Signal Detection Theory and Psychophysics. Wiley. archive.org
  • Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. pubmed
  • Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14(3), 257–262.
  • Feldman, H., & Friston, K. J. (2010). Attention, uncertainty, and free-energy. Frontiers in Human Neuroscience, 4, 215. doi
  • Fletcher, P. C., & Frith, C. D. (2009). Perceiving is believing: a Bayesian approach to explaining the positive symptoms of schizophrenia. Nature Reviews Neuroscience, 10, 48–58.
  • Adams, R. A., Stephan, K. E., Brown, H. R., Frith, C. D., & Friston, K. J. (2013). The computational anatomy of psychosis. Frontiers in Psychiatry, 4, 47. doi
  • Powers, A. R., Mathys, C., & Corlett, P. R. (2017). Pavlovian conditioning–induced hallucinations result from overweighting of perceptual priors. Science, 357, 596–600. doi
  • Pellicano, E., & Burr, D. (2012). When the world becomes “too real”: a Bayesian explanation of autistic perception. Trends in Cognitive Sciences, 16(10), 504–510.
  • Van de Cruys, S., et al. (2014). Precise minds in uncertain worlds: predictive coding in autism. Psychological Review, 121(4), 649–675.
  • Sterzer, P., et al. (2018). The predictive coding account of psychosis. Biological Psychiatry, 84(9), 634–643.
  • von Holst, E., & Mittelstaedt, H. (1950). Das Reafferenzprinzip. Naturwissenschaften, 37, 464–476.
  • Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995). An internal model for sensorimotor integration. Science, 269, 1880–1882.
  • O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 939–973. doi
  • Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11, 127–138.
  • Held, R., & Hein, A. (1963). Movement-produced stimulation in the development of visually guided behavior. J. Comparative and Physiological Psychology, 56(5), 872–876.
  • Hawkins, J., et al. (2019). A framework for intelligence and cortical function based on grid cells in the neocortex. Frontiers in Neural Circuits, 12, 121.
  • Clay, V., Leadholm, N., & Hawkins, J. (2024). The Thousand Brains Project. arXiv:2412.18354. arxiv
  • Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016). Computational psychiatry as a bridge from neuroscience to clinical applications. Nature Neuroscience, 19(3), 404–413.
  • Lappalainen, J. K., et al. (2024). Connectome-constrained networks predict neural activity across the fly visual system. Nature, 634. doi
  • Jirsa, V. K., et al. (2017). The Virtual Epileptic Patient: individualized whole-brain models of epilepsy spread. NeuroImage, 145, 377–388. doi