SMILE II: Automated sign language recognition at sentence level


Background and aims

The project builds on the groundwork of the SNSF Sinergia project SMILE I, in which an automated assessment system of manual components for DSGS at word level was developed. SMILE II extends this technology to sign language recognition and assessment of DSGS at sentence level production, including both manual and non-manual components.

Project Management
Prof. Dr.
Professor für Gebärdensprache und Partizipation bei Hörbehinderung / Leiter Bachelorstudiengang Gebärdensprachdolmetschen
tobias.haug [at]

Project Team

Katja Tissi
S. Sidler-Miserez
Financial support


Some of the challenges in the project are: (a) lack of DSGS resources (e.g. language corpora); (b) continuous sign language contains both manual information (information on the hands/arms) and non-manual information (information on the torso, head, and face), and almost all work on recognition to date has focused on manual information only; (c) lack of standardized instruments for DSGS assessment; and (d) lack of automatic methods to assess continuous sign language productions at both the manual and non-manual level.

Collaborations and methods

To achieve the ambitious goals of the project, researchers from different fields will collaborate across four different institutions:

  • Idiap Research Institute, Martigny (automated language recognition)
  • University of Teacher Education in Special Needs (HfH) (sign language assessment)
  • University of Zurich (sign language linguistics)
  • University of Surrey (UK) (sign language technology)


HfH staff, with their expertise in sign language assessment, will employ the following methods:

  • Test development of a DSGS yes/no vocabulary test, a DSGS sentence repetition test with corresponding rating scale, a DSGS translation test, and a DSGS interview with corresponding rating scale
  • Statistical analysis of test results (Classical Test Theory and Rasch analyses)
  • Rater training, studies on rater reliability, comparisons between automated and human ratings
  • Development and analysis of feedback instruments for user studies (questionnaires, focus groups)

SMILE-II in sign language

You can see a project summary in Swiss German Sign Language in the following video. You can also activate German subtitles in the video.