In the past the surgeon examined the patient's X-rays and relied on his wealth of knowledge and skill to perform the procedure. He was limited by what he could see on the x-ray, and the anatomy exposed by surgically opening the patient. 3D visualisation of the patient’s anatomy was possible with the advent of computer tomography (CT) and magnetic resonance imaging (MRI). The use of computers for stereotaxy led to the development of comprehensive frameless image-guidance systems (IGS) for surgery, as we know them today.

In conventional consultations, the physician applies the senses of sight, sound, smell and touch to make a diagnostic formulation. The remote monitoring of outpatients, in addition to telemetric transmission of vital sensor data, is increasingly based on live video communication.

Our research focus is the feasibility of a future physician - patient interface based exclusively on augmented reality video communication with embedded sensor and shared video data.

In 1995 Truppe and Ewers published worldwide first a clinical case using augmented reality in stereotactic computer assisted surgery (1). The Virtual Patient software framework was further developed to transmit intraoperative stereotactic navigation data embedded in shared video streams for remote visualization of minimal invasive surgery and second opinion advice from an expert (2).

We develop new methods to enhance the diagnostic and therapeutic value of live video imaging data. Biometric, topographic and facial recognition data is reconstructed from multi-image video sequences acquired by mobile handheld devices. The video is shared on 3G networks based on open standards such as 3GPP. The physician can view a live video or a video clip in real time during a normal voice call with the patient. Sent from one phone to the other, patient and physician can see the same video, for example from the wound area, and discuss it.

top

Research Topics

Mobile augmented reality visualisation in shared 3GPP video streams
To extract camera motion information from images and/or image sequences two distinguished processes have to be implemented: tracking and calibration. The modelling the camera’s optical features (e.g. focal length, lens distortion) is already available in the Virtual Patient framework (United States Patent 5,704,897), including the online registration of the camera to the patient’s volume (CT, MRI) or projective (X-ray) imaging data sets. The task of tracking "structure from motion" is currently under development and will be used for registration of 3-D surface models to the patient, using shared video streams for live augmented reality visualisation.

Facial expression and pain marker recognition from shared 3GPP video streams
The Facial Action Coding System (FACS) has been used, for example, to demonstrate differences between genuine and simulated pain and differences between the facial signals of suicidal and non-suicidal depressed patients. Dalton has shown that facial expressions were found to be associated with true myocardial infarction. Our aim is to develop an instrument validly describing pain and subsequently prodromal and acute symptoms of myocardial infarction, based on facial expression analysis in video sequences.

top

3-D face recognition and identification from shared 3GPP video streams

A technology called three-dimensional morphable modelling, which fits a 3-D model to a facial image taken from the side, and then turns that image into a frontal shot is considered most promising to improve nonfrontal face recognition. We develop biomechanical modelling algorithms to assist robust and pose independent face recognition, based on our software for simulation of plastic surgery procedures.

(1) Truppe M, Pongracz F, Ploder O, Wagner A, Ewers R
Interventional Video Tomography. Proceedings of Lasers in Surgery. Vol. 2395. 1995, SPIE: San Jose, CA. 150-152.

(2) Wagner A, Millesi W, Watzinger F, Truppe M, Rasse M, Enislidis G, Kermer C, Ewers R
Clinical experience with interactive teleconsultation and teleassistance in craniomaxillofacial surgical procedures. J Oral Maxillofac Surg. 1999 Dec;57(12):1413-8.

top