Classification of Signs in Sign Language using Machine Learning Methods
Thesis Supervisor: RNDr. Šimon Horvát, PhD.
The full thesis in Slovak can be found at this link.
In the bachelor thesis, we focused on classifying 25 signs from the American Sign Language alphabet, as shown in the image below:
American Sign Language gestures [1].
We used two datasets: the first in the form of raw images depicting individual gestures, and the second, which contains these images in a structured and preprocessed form. We applied both traditional machine learning methods and deep learning methods, achieving the most accurate and efficient solution using convolutional neural networks.
In addition to classification, we also focused on localizing the signing hand in the input image and constructing a a bounding box around the hand. For this task, we used a convolutional neural network with five output layers. The first layer predicted the respective letter of the alphabet, and the remaining four layers predicted the coordinates of points, which are shown as circles in the output below:
Main output of the bachelor thesis.
This model achieved a gesture classification accuracy of 96.69%, with an average coordinate prediction error of 5.18 px.
The thesis defense presentation in Slovak with more details is available at this link.
References
[1] VALLI, C., LUCAS, C., 2000. Linguistics of American sign language: An introduction. Gallaudet University Press.
