Tell Me a Story

Storytelling is a central part of human history, spanning from the oral tales of the griots to cave paintings to traveling bards. The human brain is incredibly adept at hearing or reading these stories and recreating the sensations needed to truly experience the vivid details of a masterfully-told story. This exhibit attempts to capture the user's story, related orally, and project back the regions of the brain activated by their tale. Specifically, it was designed for pre-kindergartners from Carnegie Mellon’s Children’s School who were learning about the brain.

A 3D brain mesh floats (you may recognize it from the background of this site!) as the child begins to retell their busy and exciting weekend. Perhaps they ran into Jenny at the park, or had a scary dream about a coyote (true story), or better yet adopted a puppy. The exhibit was able to pick up on a few key features: the content of their story and the pitch contours of their voice. If the child mentioned they were scared, the amygdala would be highlighted (the amygdala is involved in fear modulation). If they were a retelling a story about the park, the hippocampus would light up (hippocampus is heavily linked to spatial memory). This same simple association was done with other active verbs, pronouns, etc to map these stories to regions of the brain. A simple word embedding was used for this. For the “excitement” detection I extracted the pitch contour from the audio and looked at very simple features like the range of the pitch (the higher the pitch range the more excited the voice sounds). If you find this type of analysis exciting check out this paper I worked on 9 months after this project.

As any educator can tell you, there is something special about watching a student’s eyes widen as they grasp a concept. While I can’t pretend that these 3-5 year olds could suddenly point out where the hippocampus was with pinpoint precision, nor could I prove to you my model was that precise (it hilariously mapped a friend’s story about buying yogurt from Trader Joe’s to fear), the children were asking in-depth questions about an organ they seldom thought about before.