Bringing Storybooks to Life with Machine Learning and Coding

In 2021, in collaboration with my friends and colleagues Andrea Devore and Mariana Tamashiro, we faciliated a virtual workshop for an event called PLAYFEST. This “day of wonder” conference was hosted by the Playful Learning lab at the University of St Thomas. We were thrilled to share our process of getting creative with machine learning tools and beginner-friendly coding environments to build playful programs that recognized book covers and triggered a response on the computer such as an animation or a sound. Just the phrase, "machine learning" can evoke a sense of complexity or intimidation. However, we were excited to show how with just a little guidance, anyone can tinker with these ideas!

Here are the slides from our session which include instructions that you can follow along with:

Bring Storybooks to Life-2.pdf

Example slides:

This slide shows how we used Google's "Teachable Machine" tool to use machine learning to create a model that recognizes when I'm holding a specific storybook up to my webcam.

This slide shows what it looked like when I connected my machine learning model to a coding environment called Adacraft. Now, when I hold up specific pages from this storybook to my webcam, a special animation will play on my computer that's related to the story I am reading.