VR/AR Hackathon at MIT

The best trait about hackathons is that usually you arrive not knowing what you're going to create only knowing the timeline. After a week in Florida and a series of grueling job interviews I arrived in Boston, Mass., on Oct. 6, totally scatter-brained. I had just secured a place to stay the night before and still wasn't sure how to get to my hotel from the airport. 

To say I was a VR newbie would be an understatement. Prior to arriving in Boston,  I'd attended one VR/AR Meet Up in Chicago. I had tried VR twice and got sick both times. I really didn't see the potential or appeal of VR. I was more of an AR fan. But that weekend at MIT changed all that. 

After a day filled with workshops I had to pick five other strangers to make epic shit with. After hours of wondering around a large hall at the MIT Media Lab I finally found a group of young men who I would spend the next 36 hours with working on making the impossible possible. 

The Team

The Educator

A technologist and an educator, Jeff King taught Chemistry and Mandarin Chinese to high school students in addition to getting his master's in education technology from Harvard University. While teaching he decided to use gaming to help his students learn better. He founded EduSaga, which gives educators and learners the power to create and access interactive and immersive speaking activities to provide instant feedback through voice recognition. 

At the hackathon he explained his idea: His students studied for years but came away without any conversational language skills. Why not use VR to create an immersive environment where they could practice a foreign language in real-world situations but without the embarrassment of making common mistakes?

As a person who studied Spanish for eight years lived in South America but can only hold Spanish-language conversations with 8-year-olds I could relate. 

The Unsung Hero

It's impossible to meet Mark Kabban and not be utterly transformed. He's the founder and Executive Director of YALLA, a college prep organization that uses soccer and other youth activities to motivate refugee and immigrant youth to high achievement. This Lebanese wonder has a great vision, and even bigger heart for youth who come to the States. Studying the education and technology program at Harvard, Kabban brought the soul and wonder to our project.

His voice and insights were the guiding light for our immersive language learning app. Mark kept us focused on the user, a young boy named Eduardo, a 9-year-old from Nicaragua who's family fled to the U.S. and who is now learning English.

An actual CNN Hero, Kabban is a true jewel within in our nation, helping thousands of refugee and immigrant children to live and thrive here in America. 

The Software Engineer

Zachery Schiller is all that is wonderful about the millennial generation. A software engineer with his M.S. in Computer Science, Schiller has the programming chops you would expect. But he wasn't well versed in Unity, the main game engine that powers VR. Over the course of a weekend Schiller provided the much needed extra tech hands recording audio for our voice recognition needs as well as learning and creating Unity scenes as the same time. 

The Unity Ninja

At 20 years old, Krystian Babilinski was the veteran of our crew. A Unity developer for five years, yes he started as a teen-ager, Krystian was repeatedly interrupted experts, novices and judges alike for guidance on navigating the Unity game engine. Thank the universe he was on our side. Thanks to Krystian's skills we were able to make the speech-to-text and voice recognition parts of our application work beautifully. Not to mention his great animation skills and Unity assets procurement skills. Still it wasn't all development for Krystian. He had a real heart for our user and kept a good outlook even when us product folks were asking him to do what seemed to be the impossible. 

The UX Lead

I was the UX Lead on the project but was so excited because all the members of my team were focused on the user experience. If there ever was a tech platform that cries out for good UX it's VR. A bad user experience in VR is like being in prison: you're immersed in a sucky reality that you literally can't escape. 

I grilled Mark to gain insights on what our users would need in a language-learning app and even did a quick 30-minute survey of folks through social media who had struggled learning a foreign language. From that little design research I came away with the follow insights:

• The learning had to be natural. Most foreign language learners complained of learning words by not having conversations. Making sure the app allowed the user to engage in a conversation, even if one or two word answers were the norm was the best. 
• The learning had to be situational. We couldn't just have a script with a call and response ie., saying the word in one language and having the user repeat it in another. The user had to feel truly immersed in the scene so that he/she spoke conversationally. 
• There could be very little latency. Any pause in the app's response would break the immersive bond and therefore the conversational style. 

Finished Product

For all it's technological advancement AR/VR is all about that age-old platform media. I guess this isn't a revelation but when I reflect upon all we did in those two days designing this product it was akin to producing a television show....but a television show with some highly technical back-end work. In all we:

• Animated four different scenes
• Created for animated characters
• Wrote scripts for those characters and scenes
• Recorded those scripts
• Used two voice actors
• Integrated speech-to-text-speech recognition technology into Unity
• Connected IBM Watson, to Unity
• Videotaped the entire interaction
• Storyboarded all the scenes
• Matched educational and language pedagogy to our scripting and scene-making

Whew...that's a lot. A lot,  a lot. 

To this day that two-day hackthon was the best teaming experience I've ever had and it was totally random.