VR update: Operating room and Medical imaging VR modules

Examining the Operating room VR module

The first module is our operating room VR module, which was one of our first experiments in creating a VR environment. We’ve repurposed a number of models from our past projects to construct the room, including assets in our Until Then animation short released last July, as well as a rough liver model used in a liver transplant case.

Grabbing the liver in VR

Adjusting the overhead OR lights

Dividing the liver

The goal of the experiment was to observe how users can interact with various objects within the VR environment. These interactions range from grabbing/holding objects, manipulating lighting, to dividing the liver using a literal ‘surgical plane’ tool.

Looking down to grab the liver

Looking up to grab the lights

Looking left for surgical tools; looking right to place divided liver graft into ice bucket

One interesting observation we’ve made concerns the placement of various objects. The lights, as well as the surgical tools are placed out of the direct view of the user. While this was done mostly to mirror the actual environment in the OR, we found users were encouraged to look around their immediate surroundings, which enhances the engagement between the user and the reality we’ve placed them into.

The module currently acts as our introductory ‘tutorial’ experience for users who are unfamiliar with VR. Numerous additions are planned for the future, including clear visual feedback cues during user-object interactions, as well as additional surgical tools that perform different functions on the anatomy.

Medical imaging VR module

Previous black/white iteration of medical imaging module

Colored imaging module, with plain/venous/arterial phases

The second module is an extension of an earlier experiment, the Medical Imaging VR Module. The idea behind the module was to import CT/MRI data directly into VR, in form of volumetric data (i.e. 3D pixels, or voxels). The earlier experiment allowed for greyscale voxels to be displayed, and we’ve since started to incorporate color in an attempt to differentiate between anatomical structures.

Polished geometric models can be created in a few days' time, with clearly differentiated structures

In optimal conditions, volumetric models can be generated within hours, but with less options to isolate structures

The rationale behind utilizing voxels and volumetric data is speed - a set of CTs with slide thickness ~3cm can yield a viewable set of voxels in a matter of hours. This can be especially beneficial in visualizing patient anatomy under a tight schedule. The drawback, however, is the ability to isolate or highlight specific organs or vessel branches.

Another aim of this module was to bridge the gap between the 2D representation (flat, CT slices) of a 3D object (patient anatomy). The concept of transforming CT/MRI scans into 3D volumetric models is not novel - in fact this is already done in various software packages such as Osirix, Horos and Myrian, which we utilize in our workflow. However, these models continue to be represented on 2D screens, with interactions limited to the keyboard and mouse. We wanted to explore the experience of interacting with 3D volumetric models, while being in a 3D space.

Interacting with the 3D volumetric model

To this end, we designed the module so that the user can grab/rotate the model in 3D, as well as being able to ‘slice’ through the model in 3 different axes (coronal, axial and sagittal). The model can also be scaled up for closer inspection of anatomical features.

'Pins' to correlate between 2D/3D imaging modalities

We’ve also implemented an optional ‘pin’ system, in an attempt to bridge the different 2D and 3D imaging modalities. Users are able to place specific ‘pins’ on any given slice of the CT, and would be able to observe the pin in the 3D volumetric model, providing spatial context of the anatomy that surrounds the placed pin.