MedRoom Ecosystem

Clinical Cases is a virtual reality (VR) project that lets users — mostly medical students — go through a complete clinical interview, divided into History Taking, Physical Examination and Treatment Plan.

My challenge was to lead the design and experience for replacing the closed consultation model — a pre-written script — with an open one, where the student can talk freely about any subject with the virtual patient. The project’s structural base now deals with artificial intelligence (AI), procedural animation, natural language processing (NLP), generative models and emotion classifiers.

The entire transition to the generative model was done together with the end user. I worked on site at the institutions, acting as the interface between user, development and business. The hypotheses revolved around the student’s relationship with the metaverse, their perception of artificial intelligence, digital health, and how learning technologies and methodologies motivated a young adult — intrinsically or extrinsically — to engage with and learn from the virtual experience.

Investigating how clinical reasoning develops within traditional models was essential. At this stage I worked closely with teachers and students to understand the greatest difficulties in the learning cycle, highlighting opportunities that could be met or complemented, and deepening my own understanding of the ideal learning cycle when treating a patient.

Models and mind maps began to be proposed, reproducing the learning process along the student’s journey, and the information architecture started to be sketched out for testing.

Field work also produced the personas, reinforcing the focus on the end user and highlighting that we deal with a very specific slice of user: one under constant pressure from the institution, from family and from themselves — a student handling emotional strain at a heightened level.

Mapping the before, during and after helps reveal how the user relates to the tool. Visibility over the whole journey gave us more control over it and made possible bottlenecks more predictable — especially those tied to the new system’s reliance on voice interaction.

The final stage of understanding the problem involved a deep study of assessment practice. Artificial intelligence would now contribute to — and be responsible for — building the student’s learning. We therefore studied the skills, competencies and methods needed to construct knowledge, dividing them into three domains: cognitive, psychomotor and affective.

With the systemic problem defined and the actors and ecosystem visible, we began designing on two fronts: refining the interface and experience with the personas and their needs in mind, along with their journeys inside and outside the institution; and iterating the 3D avatar’s prompt, polishing the communication method and the feedback system.

Working with technology demands a systemic, constant gaze to avoid a few possible traps:

• Loss of perceived value: technological innovation creates demand because it differentiates you from competitors — but prioritising it as a marketing showcase can divert focus and investment away from the real value the tool can offer inside the ecosystem. Leadership that balances the dialogue between business and product is essential. Technology has no value if it isn’t integrated with the user’s pain; without perceived value and usefulness, it becomes disposable.
• Dependence on external tools: building an innovation ecosystem may require orchestrating several third-party solutions, which makes a realistic assessment of return on investment essential — the business models of external tools can multiply unforeseen costs.
• Adaptable systems: taking on a project that engages with constantly evolving technology demands a robust design, able to iterate on itself to adapt to whatever ships next. VR hardware, like ChatGPT and its versions, allowed the products to offer new possibilities and new results.

Qualitative interviews revealed that students:

• Felt safer and better prepared, seeing the metaverse and AI as a form of judgement-free training that let them better control their emotions in a supervised simulation, with reports of improved communication, clearer thinking and a sense of belonging to the technology’s potential;
• However, some students report disliking how certain physical interactions translate to the virtual world, given the haptic limits of VR — the absence of temperature and physical impact. Others report discomfort, nausea or dizziness.