December 09, 2024

Scripps Research scientists create AI that watches videos by mimicking the brain A new, more sustainable AI model recognizes visual scenes by mirroring brain processes, opening doors for applications in medical diagnostics, drug discovery and beyond.

December 09, 2024

LA JOLLA, CA Imagine an artificial intelligence (AI) model that can watch and understand moving images with the subtlety of a human brain. Now, scientists at Scripps Research have made this a reality by creating MovieNet: an innovative AI that processes videos much like how our brains interpret real-life scenes as they unfold over time.

This brain-inspired AI model, detailed in a study published in the Proceedings of the National Academy of Sciences on November 19, 2024, can perceive moving scenes by simulating how neurons or brain cells make real-time sense of the world. Conventional AI excels at recognizing still images, but MovieNet introduces a method for machine-learning models to recognize complex, changing scenes a breakthrough that could transform fields from medical diagnostics to autonomous driving, where discerning subtle changes over time is crucial. MovieNet is also more accurate and environmentally sustainable than conventional AI.

The brain doesn't just see still frames; it creates an ongoing visual narrative, says senior author Hollis Cline, PhD, the director of the Dorris Neuroscience Center and the Hahn Professor of Neuroscience at Scripps Research. Static image recognition has come a long way, but the brain's capacity to process flowing scenes like watching a movie requires a much more sophisticated form of pattern recognition. By studying how neurons capture these sequences, we've been able to apply similar principles to AI.

To create MovieNet, Cline and first author Masaki Hiramoto, PhD, a staff scientist at Scripps Research, examined how the brain processes real-world scenes as short sequences, similar to movie clips. Specifically, the researchers studied how tadpole neurons responded to visual stimuli.

Tadpoles have a very good visual system, plus we know that they can detect and respond to moving stimuli efficiently, explains Hiramoto.

Left to right: Hollis Cline, PhD, and Masaki Hiramoto, PhD. Credit: Scripps Research.

LINK:	https://www.scripps.edu/news-and-events/press-room/2024/20241209-cline...
	See more stories from scripps

North America Stories