Virtual reality for the visually impaired: VIDET

A 3-year research project (1996-1999) of:

DEIS (Dept. of Electronics, Computer Science and Systems), DIEM (Dept. of Mechanics), DM (Dept. of Mathematics),

supported by the University of Bologna and also by: ASI, CNR, Ser.In.Ar.

Project coordinator: Prof. Claudio Bonivento (LAR-DEIS)



             

What is VIDET

Acquisition and depth computation

Surface interpolation

Mechanical device

Control

Colour rendering

Bibliographic references

ich [...] sehe mit fühlendem Aug, fühle mit sehender Hand.
(J.W. v. Goethe, Römische Elegien)

I [...] see with feeling eyes, I feel with seeing hands.


What is VIDET

The VIDET (VIsual DEcoder by Touch) project aims to the design of a device converting depth information about a visual scene, into tactile information for a visually impaired user. The steps of this conversion are:

  • a camera pair (mounted on the shoulders of the user) captures two images of the scene;
  • a portable computer extracts depth information from the images and interpolates it into a surface;
  • this virtual surface, suitably scaled, is presented to the user by means of a wire-operated force-feedback robotic device (the WireMan): The user's finger is lodged in a thimble at the end of three wires coming out of a knapsack; these react to movement only when the finger hits the virtual surface.
  • The effect is that the user feels a "bas-relief" model of the scene. A most likely added performance will be colour rendering. Eventually, also global acoustical information ("virtual rain") will be added.

    Starting from an initial idea of M. Ferri, encouraged by T. Salmon Cinotti, C. Bonivento and G. Vassura, VIDET is coming into shape by a strongly interdisciplinary effort based at DEIS (directed by G. Falciasecca) in its Laboratory of Automation and Robotics (LAR). The advice and help of the APH (Handicap Project Association) of Bologna is gratefully acknowledged.


    Acquisition and depth calculation

    (T. Salmon Cinotti, L. Di Stefano, C. Lamberti, G. Paolinelli, S. Mattoccia)

    The currently devised solution

    We are now working with Ayache's algorithm for computing stereo from edges (detected by Canny's algorithm). This particular method allows the search of the solution in a pre-defined range of depth values: This fits well with the situation in VIDET, where the presence of an object is of interest only when the corresponding surface patch is near the exploring finger.

    Alternative solutions

    The option of direct matching on pixels has been discarded - at least for the moment - as too unreliable.

    Trinocular stereo - while multiplying costs and computing time - might grant the needed robustness.

    Other depth-measuring devices (e.g. laser range finders) might work better than stereo. They seem to have some drawbacks, at least in the application of interest.


    Surface interpolation

    (M. Ferri, F. Domenicucci, C. Collina)

    The currently devised solution

    A global Delaunay triangulation, constrained to contain the detected straight edges, is performed. The distance from the finger to the surface is computed along the straight line, determined by the last two sampled positions of the finger itself.

    Alternative solutions

    The hypothesis of having depth labels on all points of a Cartesian grid would allow to interpolate only locally, but this would require an intermediate interpolation; this would be both unreliable and time-consuming.

    An interpolation limited to a strip along the projection of the finger trajectory is under study.

    Later on, we plan to use a smooth interpolating surface, e.g. by B-splines.


    Mechanical device

    (G. Vassura, V. Ciavatti, S. Monti, G. Nannoni, G. Verona)

    The currently devised solution

    The WireMan (Wire Manipulator) is a robotic force-feedback, wire actuated device. It consists of a rigid frame to be carried as a knapsack; from three corners of the frame, as many pre-tensed wires converge to a thimble, lodging the user's finger. Measure of the unwinded stretches informs the system on the finger's position with respect to the reference frame of the device.

    When the finger reaches the virtual surface, the three motors exert a force - through the wires - simulating the geometrical constraint due to the surface.

    Alternative solutions

    A robotic arm placed on a shoulder, realizing the three polar coordinates, was the first choice. Drawbacks are: weight, needed power, reduced workspace, reduced resolution.


    Control

    (C. Bonivento, C. Melchiorri, A. Eusebi, L. Marconi, M. Montanari)

    The currently devised solution

    During the free space movements, the wires are maintained tensed, so that a correct measure of the finger's position can be carried out. This tension is, anyway, weak enough to allow the user a full freedom of movements.

    When the thimble is "in contact" with the virtual surface, a force, perpendicular to the surface itself, is computed. Its intensity is proportional to the penetration of the finger through the surface. This is the desired force, which would best realize the correct feeling. But this force can be generated at the thimble only if its vector lies inside the infinite pyramid having the edges along the wires. All other vectors have negative components along some of the wires, so they cannot be realized; in these cases, the interested wires are put under a minimum pre-defined tension. The net feeling - as results from experiment - is all the same realistic, and pushing against the surface yields a sliding motion along it.

    Alternative solutions

    A setup with only one active wire is under study.


    Colour rendering

    (M. Ferri, C. Collina, L. Cappelletti)

    The currently devised solution

    A triple of vibrators, applied at three fingertips, is activated at a constant frequency, at amplitudes proportional to the RGB signals.

    Experiments with seeing and with blind subjects have been performed with success.

    Alternative solutions

    The vibrators were initially applied to the three phalanxes of the same finger. This turned out to be a more fatiguing and confusing setup.

    An acoustical signal might either replace or integrate the tactile one. In this case, it might be at three fixed frequencies, with varying amplitudes, or - more likely - at varying frequencies.


    Bibliographical references

    Di Stefano, L., and Paolinelli, G., Stereo research within the VIDET project, submitted.

    Cappelletti, L., Ferri, M., and Nicoletti, G., Vibrotactile colour rendering for the visually impaired within the VIDET project, submitted.

    Bonivento, C., Di Stefano, L., Ferri, M., Melchiorri, C., and Vassura, G., VIDET: un sistema per videolesi basato sulla resa tattile di ambienti virtuali 3D, submitted.

    Melchiorri, C., Montanari, M., and Vassura, G., Control strategies for a defective, wire-based, haptic interface, Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Grenoble, Sept. 8-12, 1997.

    Bonivento, C., Eusebi, A., Melchiorri, C., Montanari, M., and Vassura, G., WireMan: A portable wire manipulator for touch-rendering of bas-relief virtual surfaces, Proc. ICAR '97, Monterey, July 7-9, 1997.

    Bonivento, C., Di Stefano, L., Ferri, M., Melchiorri, C., and Vassura, G., The VIDET project: Basic ideas and early results, Proc. ICAR '97, Monterey, July 7-9, 1997.

    Di Stefano, L., Eusebi, A., Ferri., M., Melchiorri, C., Montanari, M., and Vassura, G., A robotic system for visually impaired people based on stereo-vision and force-feedback, Proc. IARP Int. Workshop on Medical Robots, Vienna, October, 1-2, 1996.


    Vision Mathematics home page
    L.A.R. home page


    Last revision: 30 December 1997