Augmented Reality and Medicine with Microsoft Hololens

In May of 2016, I entered the Seattle Microsoft Hololens Hackathon.  We had 44 hours to build a project with Unity and Microsoft Hololens.  The tech of our project was composed of C#, Javascript, and OSC protocol.  I had met Ryan James at the previous VR Hackathon and was really interested in his work with medicine, learning and simulations.  I was really happy to join his team and excited to design sounds for the project and work on my Unity chops.

The challenge was to create a simulation that trained medical practitioners how to guide a catheter into the right side of a patient's heart, then make a precise puncture in the middle of the heart to gain access to the left side of the heart. We wanted to understand how augmented reality could improve this very important and widely used technique.

Transeptal Puncture and Catheter Integration Simulation

Transeptal Puncture and Catheter Integration Simulation

We set out to create a simulation that will provide a better look and practice, with a hologram of the heart.  Dozens of xrays are usually conducted during the procedure, injecting dyes into the tissue for visual feedback.  Part of the holographic ability was to display a 2D close up and 3D view simultaneously.  This was an early team UI decision. We also knew the Hololens would be great for monitoring vitals and managing patient information privately.  

Ahmad Aljadaan and I set out to design the audio cues early on so that our voice actor, Bridget Swirski could record them.  We quickly began to realize the importance of these cues in the user interface.  The audio cues guide most of the experience, creating feedback for the user when they are on and off track in the procedure, and giving them directions as they guide the catheter and needle through the inside of the heart and arteries.  We chose a female voice, as suggested in a lightning talk by the Hololens spatial audio design team.  The human voice has excellent spatial recognition in the AR environment.

These audio cues helped refine our approach into the experience storyboard:  

We knew the end user would need a granular method of controlling the catheter and needle.  OSC seemed like a great choice since it's a low latency TCP protocol used for musical instrumentation.  It works with iOS and Android already, and I found some packages for Unity for us to modify.  It took a few tweaks and a lesson in switch loops in Javascript by Ryan, and we got it going.  In hindsight, I might purchase one of the more refined OSC kits from the Unity Asset Store in the future to save time in hackathon-type situations.

TouchOSC Custom Controller Interface I designed for Single-Hand Operation (accelerometer control was a little squirrelly for surgery)

TouchOSC Custom Controller Interface I designed for Single-Hand Operation (accelerometer control was a little squirrelly for surgery)

We crunched and coded to the very beginning of our presentation.  The scope of our project was a little big for the weekend, and we did a lot.  Sometimes the scope of your project outgrows the hackathon and it certainly did this time. 

My team was awesome.  Check out these rock stars:

Ryan James: Lead Developer

Ahmad Aljadaan: Developer, UX

Mark Laughery: Scene Design, UX

Jigesh Parekh: OSC Integration, 2D Design, Presentation

Andrew Luck (me): Sound Design, UI, OSC Integration, Presentation Design, Video Editing, 2D Motion