Essential Vision: ETH Zurich - using augmented reality in training healthcare professionals

Training medical professionals comes with high costs, starting from access to training equipment and ending with knowledge transfer from teachers and more experienced specialists. According to a paper published in the Journal of Health Services Research, training a general practice doctor costs up to $451,000 per completing student (or $18,400 per year of practice). The cost was calculated using university fees, the cost of internships, and payments and was adjusted for student drop-out rates.

The better the training and education, the more skilled doctors will be and their work will further contribute to society and help save and improve lives. This is precisely where Tooploox wishes to be in helping with our technical skills and expertise.

Training medical staff is the ideal spot where Tooploox and ETH Zurich have found a way to improve people’s lives through technology.

The partner

ETH Zurich, or the Swiss Federal Institute of Technology in Zurich, Switzerland, is a highly renowned academic research institute that has many renowned affiliated scientists, including Albert Einstein and John von Neumann, among others – the institution boasts 22 Nobel prize laureates. In the 2022 edition of QS World University Rankings, ETH Zurich was ranked 8th in the world and 4th in Europe.

Our immediate project partner was The Zurich Center for Experimental and Clinical Imaging Technologies (EXCITE). EXCITE is a joint competence center of ETH Zurich and the University of Zurich under the umbrella of Hochschulmedizin Zürich (HMZ). Its objective is to provide a broad platform for imaging solutions across length scales, from single atoms to anatomical features. The focus of the center is on the development of biomedical imaging solutions, which explicitly includes their translation into clinical research and practice.

The challenge

Modern medicine, including training, has a mind-boggling range of tools, starting with super-modern MRI scans, computerized tomography, and AI-aided diagnosis. On the other side is the centuries-long tradition of dissecting room lessons and analyzing dissected organs.

When learning anatomy, dissection rooms and anatomy classes with dissected organs are necessary yet rarely sufficient. The human body is a genuine mechanism, with tiny structures that are hard to examine or sometimes even to see, especially in a crowded classroom. Also, in the dissection room only dead specimens are shown with postmortem changes which affect the structure and color of organs. Their method of preservation also affects the overall look, not to mention that a living human is always in motion, with blood flowing and organs perpetually in motion.

EXCITE accepted the challenge of tackling the status quo, and Tooploox has come up with a solution.

Our work

The key partner in this project, EXCITE, provided Tooploox with data harvested using the Computer Tomography scans of living patients. These scans are of immense diagnostic value, yet they require a skilled radiologist’s interpretation for each image.

How CT works

Computed tomography is ever-improving technique that is a direct descendant of x-ray scans. Yet there are several important differences one needs to consider.

The CT scan is done while the patient is lying down and the image shows a horizontal slice while an X-ray shows a patient frontally by default.
The complete scan is done by taking a series of images, effectively producing a short film that can be further processed into a 3d shape.
Because CT only shows how well a tissue absorbs X rays in grayscale, natural colors need to be applied either manually or using sophisticated software.

CT-produced image data of a particular organ is then extracted using computer software and later improved upon manually by radiologists, who may be rather unfamiliar with the manual adjustment step. Later, these 3D images can be made into models that can be viewed on a computer screen.

3D models of internal organs and structures were delivered by the EXCITE team and ready to be used in a novel, augmented reality geared for learning.

VR and AR

Seeing a 3D model on a computer screen is cool and can deliver much more interesting information for either doctor or student. Yet this is not the most convenient way to interact with a 3D model. Also, having a model displayed on a computer screen limits the interactivity of the lesson and builds unnecessary barriers.

The Tooploox team has used HoloLens, a virtual reality device delivered by Microsoft, to build a software solution that delivers an Augmented Reality learning experience in medical science training. The device displays the 3D image of a selected organ model levitating in front of the user.

The effect was delivered by converting the 3D model of an organ into a Unity format, which is supported by HoloLens. All the software framework needed to accomplish this was delivered by Tooploox, in cooperation with CatAstopheGames (CAG), our long-standing partner and AR developer.

These included tools to control the learning environment, the interface, and all the backend details that ensure the smooth working of the solution.

Depending on the user’s role (teacher or student), the displayed organs can be rotated, enlarged, or highlighted when needed.

How does it work

A 3D headset scans the room to set the positioning of the 3D model in space, for example, to place it above a table or in front of the blackboard in a classroom. One of the headsets is the server (or teacher), through which the operator has full control of the display and can manipulate the object at will.

The augmented reality approach reduces the nausea or confusion often experienced when participating in a full virtual reality experience. The 3D model is just another object in the room, not in an entirely simulated environment and so there are no problems with any slight delays in the display.

The teacher can add or remove displayed objects or load animations if there are any prepared. The manipulation of objects can be done by hand in the “Minority Report” manner.

The result

The model output takes the form of a 3D heart levitating in the middle of the classroom, with all the tiny cardiac morphology details visible to the viewers. The heart display moves in the way the organ behaves while in a patient’s body (or not, if the animation is paused by the teacher). The students can easily rotate the organ and look through the aorta to see the heart valves working.

This comes with numerous advantages in the education process:

The shape and work of a particular organ

Without the support of modern technologies, seeing a particular organ at work is a rare opportunity, inaccessible without access to a willing patient and an operating theater or MRI/CT scanner. This Tooploox-delivered technology is available anywhere the student has access to a Hololens device.
Its context among the rest of the organs

It is possible to enrich the database of 3D shapes with new organs or entire systems to see how they cooperate with each other. This provides an unseen opportunity to watch how the extremely complicated system of the human body actually works. The key limitation is access to a library of 3D shapes, which is continually in development by ETH Zurich’s staff.
Organ comparison

The system can also show changes in an organ due to surgeries or pathological processes – all accessible in detail, possible to rotate or see-through. For example, it is possible to see a heart with its original valves alongside its new artificial valves or a knee both before and after surgery. Again – the key is in data preparation and access to a library of shapes.
Shareability

The lesson does not need to be conducted in a single space. As long as the student has access to a Hololens device, he or she can participate in the lesson and reap all the benefits.

More about the project can be found on the Essential Vision website.

Summary

This platform is just the beginning and with a growing library of shapes, it will find fresh applications in training new doctors to greater mastery of human anatomy.

Let our specialists solve the problems and tackle the challenges that hold you from conquering the world.

Let’s talk