Arthur in Mixed Reality

Arthur XR

Tap 4 Technologies x Arthur App Powered by Orlando Health

June 2024 - July 2024

Overview

Arthur in Mixed Reality is concept application that reimagines the existing iPad-based Arthur App for a mixed reality experience.

The Arthur App is an iOS app originally designed to help non-verbal hospital patients communicate their thoughts, pain, and emotions with staff and family. This new version leverages mixed reality to bring those same capabilities into a 3D space, enhancing accessibility and engagement.

This project showcases the potential of immersive technology like XR to transform healthcare, providing a glimpse into how mixed reality can make patient provider interactions more intuitive, personal, and effective.

What I Contributed

I was the lead developer for the application. I was in charge of handling user interaction, core systems, and application loop. I did so by following mock-up designs and use case diagrams provided by designers. Below is a list of the main parts of the project I developed.

Chat Messaging System

I created a chat system inspired by the designs of WhatsApp and other instant message providers. Users can press buttons to submit predetermined messages, or type their own unique message with a virtual keyboard from Microsoft's Mixed Reality Toolkit.

Text-To-Speech

I implemented Text-To-Speech using Vivox services so that any message the user submits will be played out loud.

Cohesion + Immersion Features

Managed sound effects and music using Unity's Audio Mixer to provide background music and audio feedback of user interaction.
Created zoom in/out effects for transitioning between panels using DOTween.
Created fade in/out effects for transitioning between logos in the Intro Sequence using DOTween.
Handled button interaction visualization to follow the user's hand when it is in close proximity to the button.

Encountered Problems and Solutions

Panel Transitions Would Not Open New Panel

System Overview: One feature of the app is that when a button such as Symptoms or I Feel... is selected, it would open a new panel with more options based on the given category. We wanted to create a smooth transition between the two panels by scaling down the current panel, then opening the new panel. To do this, I used a coroutine to handle scaling over time. So the general process was the following:

Scale the current panel to 0
Deactivate the current panel
Activate the new panel
Scale the new panel to 1

Problem: When I pushed my changes, others reported that they could no longer open the new panel when a button is selected. I did not test my changes prior to pushing because I have worked with coroutines many times before.

Solution: The problem has to do with how coroutines behave. Coroutines actually stop their execution when the game object is deactivated. So after step 2, the coroutine will stop its execution, meaning steps 3 and 4 would not execute. Therefore, the current panel would close, but the new panel would not be activated or scaled.

To solve this problem, I modified the coroutine by reordering the steps listed above:

Scale the current panel to 0
Activate the new panel
Scale the new panel to 1
Deactivate the current panel

Therefore, deactivating the current panel would be the last executed line of code, and the new panel would open.

What I Learned: Even though I have used some sort of method/technique many times before, I do not know if it will work in all cases. I need to test everything, as one case can completely break my understanding of a system.