| Abstrak/Abstract |
Total communication in sign language involves hand gestures as manual signs and facial expressions as non- manual signs. Both types of signs have an important role in communication. Research related to sign language translators focuses more on manual signs. Ignoring the non-manual components causes the translation of spoken language to be less precise because of the loss of information from the signer’s facial emotions. Indonesian Sign Language (SIBI) is different from other sign languages. SIBI follows the rules of the Indonesian language, which consist of structure and syntax. This paper proposes a framework for sign language translation composed of manual and non-manual components. Two propositions will be tested and contribute to this research. The first proposition is related to the use of sign gesture detection and segmentation methods using optical flow-based methods, and the second proposition is associated with the use of STMC, Facial Expression Recognition, and Transformer methods at the stage of recognition and translation of sign language. The proposed work arrangement is expected to improve performance measured by WER, BLEU, and METEOR and reduce computational load as measured by processing time. |
