SMILE II: Automated sign language recognition at sentence level

Category Project

Ausgangslage und Ziele

The project builds on the groundwork of the SNSF Sinergia project SMILE I, in which an automated assessment system of manual components for DSGS at word level was developed. SMILE II extends this technology to sign language recognition and assessment of DSGS at sentence level production, including both manual and non-manual components.

Project Management

Tobias Haug Title Prof. Dr.


Professor für Gebärdensprache und Partizipation bei Hörbehinderung / Leiter Bachelor Gebärdensprachdolmetschen


  • Duration
  • Neue Projektnummer

Project Team

Financial support


Some of the challenges in the project are: (a) lack of DSGS resources (e.g. language corpora); (b) continuous sign language contains both manual information (information on the hands/arms) and non-manual information (information on the torso, head, and face), and almost all work on recognition to date has focused on manual information only; (c) lack of standardized instruments for DSGS assessment; and (d) lack of automatic methods to assess continuous sign language productions at both the manual and non-manual level.

Collaborations and methods

To achieve the ambitious goals of the project, researchers from different fields will collaborate across four different institutions:

  • Idiap Research Institute, Martigny (automated language recognition)
  • University of Teacher Education in Special Needs (HfH) (sign language assessment)
  • University of Zurich (sign language linguistics)
  • University of Surrey (UK) (sign language technology)

HfH staff, with their expertise in sign language assessment, will employ the following methods:

  • Test development of a DSGS yes/no vocabulary test, a DSGS sentence repetition test with corresponding rating scale, a DSGS translation test, and a DSGS interview with corresponding rating scale
  • Statistical analysis of test results (Classical Test Theory and Rasch analyses)
  • Rater training, studies on rater reliability, comparisons between automated and human ratings
  • Development and analysis of feedback instruments for user studies (questionnaires, focus groups)

SMILE-II in sign language

You can see a project summary in Swiss German Sign Language in the following video. You can also activate German subtitles in the video.


  • Haug, T., & Holzknecht, F.
    Using automatic sign language recognition for sentence-level assessment of Swiss German Sign Language - SMILE-II
    Language Testing Research Colloquium 2020,
  • Haug, T., & Tissi, K.
    Modalitätsspezifische Aspekte von Gebärdensprachen: Lernen, Testen und Ressourcen.
    Ringvorlesung Universität Bern, Institut für Allgemeine Sprachwissenschaften,
    Bern, Schweiz.
  • Tornay, S., Nanchen, A., Battisti, A., Holzknecht, F., Tarigopula, N., Mendez Maldonado, O., Camgöz, N. C., Razavi, M., Tissi, K., Sidler-Miserez, S., Boyes Bream, P., Ebling, S., Haug, T., Bowden, R., & Magimai-Doss, M.
    Web SMILE demo: a web application providing automated feedback on sign language vocabulary production
    44th Language Testing and Research Colloquium: Language Assessment for a Global, Digital, and More Equitable Era,
    New York City, USA.