PopSign: Sign Language Learning Games and Technologies

Goals

This project aims to develop games, toolkits, and other technologies that help deaf children and their parents acquire language skills in American Sign Language. Integral components of these efforts would be enhancing students’ knowledge and skills as part of long-term, diverse, interdisciplinary team-based project development (sense of competence with industry standards in CS/ML processes/procedures, project/time-management skills), understanding of the possibilities for development with new technologies (ML, LLMs).

Issues Involved or Addressed

American Sign Language (ASL) Background: 95% of deaf children are born to hearing parents, and most of those parents never learn enough sign language to teach their children. This puts them at high risk of suffering from language deprivation syndrome, which has emotional, mental, and cognitive effects that can be irreversible. For example, short-term memory skills are learned from acquiring a language, and many deaf children enter school with a short-term memory of fewer than 3 items, much less than hearing children of hearing parents or Deaf children of Deaf parents. Our systems address this problem from two different angles. PopSignAI teaches parents ASL vocabulary, since parents are the main vehicle through which children learn language. CopyCat improves children’s short-term memory skills through ASL games. Technological challenges: We aim to create an extensible framework for sign input that can be integrated into a wide variety of mobile apps and games, and develop a collection of games that address the learning needs of young and adult learners. To do so, we create datasets of ASL/other sign languages, used to develop Sign Language Recognition (SLR) systems for a variety of contexts and use cases. We can then extend these technologies to the broader, international population of sign language users worldwide.

Partners/Sponsors

Google; Center on Access Technology at the National Technical Institute for the Deaf (NTID) and RIT;D-PAN Deaf Professional Arts Network