Machine translation of sign languages
This article's use of external links may not follow Wikipedia's policies or guidelines. (December 2021) |
The machine translation of sign languages has been possible, albeit in a limited fashion, since 1977. When a research project successfully matched English letters from a keyboard to ASL manual alphabet letters which were simulated on a robotic hand. These technologies translate
Limitations
Sign language translation technologies are limited in the same way as spoken language translation. None can translate with 100% accuracy. In fact, sign language translation technologies are far behind their spoken language counterparts. This is, in no trivial way, due to the fact that signed languages have multiple articulators. Where spoken languages are articulated through the vocal tract, signed languages are articulated through the hands, arms, head, shoulders, torso, and parts of the face. This multi-channel articulation makes translating sign languages very difficult. An additional challenge for sign language MT is the fact that there is no formal written format for signed languages. There are notations systems but no writing system has been adopted widely enough, by the international Deaf community, that it could be considered the 'written form' of a given sign language. Sign Languages then are recorded in various video formats. There is no gold standard
History
The history of automatic sign language translation started with the development of hardware such as finger-spelling robotic hands. In 1977, a finger-spelling hand project called RALPH (short for "Robotic Alphabet") created a robotic hand that can translate alphabets into finger-spellings.[2] Later, the use of gloves with motion sensors became the mainstream, and some projects such as the CyberGlove and VPL Data Glove were born.[3] The wearable hardware made it possible to capture the signers' hand shapes and movements with the help of the computer software. However, with the development of computer vision, wearable devices were replaced by cameras due to their efficiency and fewer physical restrictions on signers.[3] To process the data collected through the devices, researchers implemented neural networks such as the Stuttgart Neural Network Simulator[4] for pattern recognition in projects such as the CyberGlove. Researchers also use many other approaches for sign recognition. For example, Hidden Markov Models are used to analyze data statistically,[3] and GRASP and other machine learning programs use training sets to improve the accuracy of sign recognition.[5] Fusion of non-wearable technologies such as cameras and Leap Motion controllers have shown to increase the ability of automatic sign language recognition and translation software.[6]
Technologies
VISICAST
http://www.visicast.cmp.uea.ac.uk/Visicast_index.html
eSIGN project
http://www.visicast.cmp.uea.ac.uk/eSIGN/index.html
The American Sign Language Avatar Project at DePaul University
Spanish to LSE
- López-Ludeña, Verónica; San-Segundo, Rubén; González, Carlos; López, Juan Carlos; Pardo, José M. (2012). "Methodology for developing a Speech into Sign Language Translation System in a New Semantic Domain" (PDF). )
SignAloud
SignAloud is a technology that incorporates a pair of gloves made by a group of students at University of Washington that transliterate[7] American Sign Language (ASL) into English.[8] In February 2015 Thomas Pryor, a hearing student from the University of Washington, created the first prototype for this device at Hack Arizona, a hackathon at the University of Arizona. Pryor continued to develop the invention and in October 2015, Pryor brought Navid Azodi onto the SignAloud project for marketing and help with public relations. Azodi has a rich background and involvement in business administration, while Pryor has a wealth of experience in engineering.[9] In May 2016, the duo told NPR that they are working more closely with people who use ASL so that they can better understand their audience and tailor their product to the needs of these people rather than the assumed needs.[10] However, no further versions have been released since then. The invention was one of seven to win the Lemelson-MIT Student Prize, which seeks to award and applaud young inventors. Their invention fell under the "Use it!" category of the award which includes technological advances to existing products. They were awarded $10,000.[11][12]
The gloves have sensors that track the users hand movements and then send the data to a computer system via
ProDeaf
ProDeaf (WebLibras)[14] is a computer software that can translate both text and voice into Portuguese Libras (Portuguese Sign Language) "with the goal of improving communication between the deaf and hearing."[15] There is currently a beta edition in production for American Sign Language as well. The original team began the project in 2010 with a combination of experts including linguists, designers, programmers, and translators, both hearing and deaf. The team originated at Federal University of Pernambuco (UFPE) from a group of students involved in a computer science project. The group had a deaf team member who had difficulty communicating with the rest of the group. In order to complete the project and help the teammate communicate, the group created Proativa Soluções and have been moving forward ever since.[16] The current beta version in American Sign Language is very limited. For example, there is a dictionary section and the only word under the letter 'j' is 'jump'. If the device has not been programmed with the word, then the digital avatar must fingerspell the word. The last update of the app was in June 2016, but ProDeaf has been featured in over 400 stories across the country's most popular media outlets.[17]
The application cannot read sign language and turn it into word or text, so it only serves as a one-way communication. Additionally, the user cannot sign to the app and receive an English translation in any form, as English is still in the beta edition.
Kinect Sign Language Translator
Since 2012, researchers from the Chinese Academy of Sciences and specialists of deaf education from Beijing Union University in China have been collaborating with Microsoft Research Asian team to create Kinect Sign Language Translator.[18] The translator consists of two modes: translator mode and communication mode. The translator mode is capable of translating single words from sign into written words and vice versa. The communication mode can translate full sentences and the conversation can be automatically translated with the use of the 3D avatar. The translator mode can also detect the postures and hand shapes of a signer as well as the movement trajectory using the technologies of machine learning, pattern recognition, and computer vision. The device also allows for reciprocal communication because the speech recognition technology allows the spoken language to be translated into the sign language and the 3D modeling avatar can sign back to the deaf people.[19]
The original project was started in China based on translating Chinese Sign Language. In 2013, the project was presented at Microsoft Research Faculty Summit and Microsoft company meeting.[20] Currently, this project is also being worked by researchers in the United States to implement American Sign Language translation.[21] As of now, the device is still a prototype, and the accuracy of translation in the communication mode is still not perfect.
SignAll
SignAll
MotionSavvy
MotionSavvy[26] was the first sign language to voice system. The device was created in 2012 by a group from Rochester Institute of Technology / National Technical Institute for the Deaf and "emerged from the Leap Motion accelerator AXLR8R."[27] The team used a tablet case that leverages the power of the Leap Motion controller. The entire six person team was created by deaf students from the schools deaf-education branch.[28] The device is currently one of only two reciprocal communication devices solely for American Sign Language. It allows deaf individuals to sign to the device which is then interpreted or vice versa, taking spoken English and interpreting that into American Sign Language. The device is shipping for $198. Some other features include the ability to interact, live time feedback, sign builder, and crowdsign.
The device has been reviewed by everyone from technology magazines to Time. Wired said, "It wasn't hard to see just how transformative a technology like [UNI] could be" and that "[UNI] struck me as sort of magical."Katy Steinmetz at TIME said, "This technology could change the way deaf people live." Sean Buckley at Engadget mentioned, "UNI could become an incredible communication tool."
References
- S2CID 5525834. Archived from the original(PDF) on 2021-01-10. Retrieved 2020-05-04.
- PMID 7965881.
- ^ PMID 16192405.
- S2CID 18434944.
- )
- PMID 32917024.
- ^ "What is the difference between translation and transliteration". english.stackexchange.com. Retrieved 2017-04-06.
- ^ "SignAloud". Archived from the original on 2020-09-21. Retrieved 2017-02-28.
- ^ "Thomas Pryor and Navid Azodi | Lemelson-MIT Program". lemelson.mit.edu. Archived from the original on 2020-09-21. Retrieved 2019-07-04.
- ^ a b "These Gloves Offer A Modern Twist On Sign Language". All Tech Considered. NPR. 17 May 2016.
- ^ "Collegiate Inventors Awarded Lemelson-MIT Student Prize". Lemelson-MIT Program. Archived from the original on 2021-01-13. Retrieved 2017-03-09.
- ^ "UW undergraduate team wins $10,000 Lemelson-MIT Student Prize for gloves that translate sign language". University of Washington. 2016-04-12. Retrieved 2017-04-09.
- ^ "Nonmanual markers in American Sign Language (ASL)". www.lifeprint.com. Retrieved 2017-04-06.
- ^ "ProDeaf". prodeaf.net. Archived from the original on 2021-03-12. Retrieved 2017-04-09.
- ^ "ProDeaf". www.prodeaf.net. Retrieved 2017-03-09.
- ^ "ProDeaf". www.prodeaf.net. Retrieved 2017-03-16.
- ^ "ProDeaf Tradutor para Libras on the App Store". App Store. Retrieved 2017-03-09.
- ^ Chen, Xilin; Li, Hanjing; Pan, Tim; Tansley, Stewart; Zhou, Ming. "Kinect Sign Language Translator expands communication possibilities" (PDF). Microsoft Research Connections. Archived from the original (PDF) on 29 March 2014.
- )
- ^ "Kinect Sign Language Translator". Microsoft. 29 October 2013.
- S2CID 5488882.
- ^ "SignAll. We translate sign language. Automatically". www.signall.us. Archived from the original on 2021-02-02. Retrieved 2017-04-09.
- ^ "Dolphio | Unique IT Technologies". www.dolphio.hu. Retrieved 2017-04-06.
- ^ "SignAll. We translate sign language. Automatically". www.signall.us. Archived from the original on 2021-02-02. Retrieved 2017-03-09.
- ^ "Fort Bend Christian Academy American Sign Language Program Pilots New Technology | Fort Bend Focus Magazine". Retrieved 2023-08-08.
- ^ "MotionSavvy UNI: 1st sign language to voice system". Indiegogo. Retrieved 2017-03-09.
- ^ "Rochester Institute of Technology (RIT)". Rochester Institute of Technology (RIT). Retrieved 2017-04-06.
- ^ Tsotsis, Alexia (6 June 2014). "MotionSavvy Is A Tablet App That Understands Sign Language". TechCrunch. Retrieved 2017-04-09.