Essential Vision: ETH Zurich - using augmented reality in training healthcare professionals
Training medical professionals comes with high costs, starting from access to training equipment and ending with knowledge transfer from teachers and more experienced specialists. According to a paper published in the Journal of Health Services Research, training a general practice doctor costs up to $451,000 per completing student (or $18,400 per year of practice). The cost was calculated using university fees, the cost of internships, and payments and was adjusted for student drop-out rates.
The better the training and education, the more skilled doctors will be and their work will further contribute to society and help save and improve lives. This is precisely where Tooploox wishes to be in helping with our technical skills and expertise.
Training medical staff is the ideal spot where Tooploox and ETH Zurich have found a way to improve people’s lives through technology.
ETH Zurich, or the Swiss Federal Institute of Technology in Zurich, Switzerland, is a highly renowned academic research institute that has many renowned affiliated scientists, including Albert Einstein and John von Neumann, among others – the institution boasts 22 Nobel prize laureates. In the 2022 edition of QS World University Rankings, ETH Zurich was ranked 8th in the world and 4th in Europe.
Our immediate project partner was The Zurich Center for Experimental and Clinical Imaging Technologies (EXCITE). EXCITE is a joint competence center of ETH Zurich and the University of Zurich under the umbrella of Hochschulmedizin Zürich (HMZ). Its objective is to provide a broad platform for imaging solutions across length scales, from single atoms to anatomical features. The focus of the center is on the development of biomedical imaging solutions, which explicitly includes their translation into clinical research and practice.
Modern medicine, including training, has a mind-boggling range of tools, starting with super-modern MRI scans, computerized tomography, and AI-aided diagnosis. On the other side is the centuries-long tradition of dissecting room lessons and analyzing dissected organs.
When learning anatomy, dissection rooms and anatomy classes with dissected organs are necessary yet rarely sufficient. The human body is a genuine mechanism, with tiny structures that are hard to examine or sometimes even to see, especially in a crowded classroom. Also, in the dissection room only dead specimens are shown with postmortem changes which affect the structure and color of organs. Their method of preservation also affects the overall look, not to mention that a living human is always in motion, with blood flowing and organs perpetually in motion.
EXCITE accepted the challenge of tackling the status quo, and Tooploox has come up with a solution.
The key partner in this project, EXCITE, provided Tooploox with data harvested using the Computer Tomography scans of living patients. These scans are of immense diagnostic value, yet they require a skilled radiologist’s interpretation for each image.
How CT works
Computed tomography is ever-improving technique that is a direct descendant of x-ray scans. Yet there are several important differences one needs to consider.
CT-produced image data of a particular organ is then extracted using computer software and later improved upon manually by radiologists, who may be rather unfamiliar with the manual adjustment step. Later, these 3D images can be made into models that can be viewed on a computer screen.
3D models of internal organs and structures were delivered by the EXCITE team and ready to be used in a novel, augmented reality geared for learning.
VR and AR
Seeing a 3D model on a computer screen is cool and can deliver much more interesting information for either doctor or student. Yet this is not the most convenient way to interact with a 3D model. Also, having a model displayed on a computer screen limits the interactivity of the lesson and builds unnecessary barriers.
The Tooploox team has used HoloLens, a virtual reality device delivered by Microsoft, to build a software solution that delivers an Augmented Reality learning experience in medical science training. The device displays the 3D image of a selected organ model levitating in front of the user.
The effect was delivered by converting the 3D model of an organ into a Unity format, which is supported by HoloLens. All the software framework needed to accomplish this was delivered by Tooploox, in cooperation with CatAstopheGames (CAG), our long-standing partner and AR developer.
These included tools to control the learning environment, the interface, and all the backend details that ensure the smooth working of the solution.
Depending on the user’s role (teacher or student), the displayed organs can be rotated, enlarged, or highlighted when needed.
How does it work
A 3D headset scans the room to set the positioning of the 3D model in space, for example, to place it above a table or in front of the blackboard in a classroom. One of the headsets is the server (or teacher), through which the operator has full control of the display and can manipulate the object at will.
The augmented reality approach reduces the nausea or confusion often experienced when participating in a full virtual reality experience. The 3D model is just another object in the room, not in an entirely simulated environment and so there are no problems with any slight delays in the display.
The teacher can add or remove displayed objects or load animations if there are any prepared. The manipulation of objects can be done by hand in the “Minority Report” manner.
The model output takes the form of a 3D heart levitating in the middle of the classroom, with all the tiny cardiac morphology details visible to the viewers. The heart display moves in the way the organ behaves while in a patient’s body (or not, if the animation is paused by the teacher). The students can easily rotate the organ and look through the aorta to see the heart valves working.
This comes with numerous advantages in the education process:
The shape and work of a particular organ
Without the support of modern technologies, seeing a particular organ at work is a rare opportunity, inaccessible without access to a willing patient and an operating theater or MRI/CT scanner. This Tooploox-delivered technology is available anywhere the student has access to a Hololens device.
Its context among the rest of the organs
It is possible to enrich the database of 3D shapes with new organs or entire systems to see how they cooperate with each other. This provides an unseen opportunity to watch how the extremely complicated system of the human body actually works. The key limitation is access to a library of 3D shapes, which is continually in development by ETH Zurich’s staff.
The system can also show changes in an organ due to surgeries or pathological processes – all accessible in detail, possible to rotate or see-through. For example, it is possible to see a heart with its original valves alongside its new artificial valves or a knee both before and after surgery. Again – the key is in data preparation and access to a library of shapes.
The lesson does not need to be conducted in a single space. As long as the student has access to a Hololens device, he or she can participate in the lesson and reap all the benefits.
This platform is just the beginning and with a growing library of shapes, it will find fresh applications in training new doctors to greater mastery of human anatomy.