The challenge was to create a simulation that trained medical practitioners how to guide a catheter into the right side of a patient's heart, then make a precise puncture in the middle of the heart to gain access to the left side of the heart. We wanted to understand how augmented reality could improve this very important and widely used technique.
We set out to create a simulation that will provide a better look and practice, with a hologram of the heart. Dozens of xrays are usually conducted during the procedure, injecting dyes into the tissue for visual feedback. Part of the holographic ability was to display a 2D close up and 3D view simultaneously. This was an early team UI decision. We also knew the Hololens would be great for monitoring vitals and managing patient information privately.
Ahmad Aljadaan and I set out to design the audio cues early on so that our voice actor, Bridget Swirski could record them. We quickly began to realize the importance of these cues in the user interface. The audio cues guide most of the experience, creating feedback for the user when they are on and off track in the procedure, and giving them directions as they guide the catheter and needle through the inside of the heart and arteries. We chose a female voice, as suggested in a lightning talk by the Hololens spatial audio design team. The human voice has excellent spatial recognition in the AR environment.
These audio cues helped refine our approach into the experience storyboard:
We crunched and coded to the very beginning of our presentation. The scope of our project was a little big for the weekend, and we did a lot. Sometimes the scope of your project outgrows the hackathon and it certainly did this time.