Based on speech-to-text, this software allows a variety of hand gestures to be recorded as input and outputs it into text!
Inspiration
Our primary inspiration was the current speech to text system, it is widely used and hence very developed and effective. Unfortunately many individuals who lack the ability to speak often cannot benefit from these advancements. This is why we have decided to contribute to the space by implementing an accessible app which is capable of translating a variety of hand gestures/sign language into text.
How we built it
We used tensorflow in order to make our application understand and track the joint locations and movements. On the basis of the users joint positions the system is capable of identifying and naming these gestures.
Challenges we ran into
The biggest problem was to train the model using python language where we also had to convert it into a tensorflow.js compatible model
Accomplishments that we’re proud of
Successfully engaging video as an input which can then be processed into useful data in the form of nameable gestures.
What we learned
The process of training models and implementing them in a program.
What’s next for Sign to Text
As we improve our data models, additional functionalities and definitions are planned to be added. These could include the implementation of seperate use-cases such as a text to braille or text to morse and vice versa.
Built With
- javascript
- machine-learning
- next.js
- react.js
- tensorflow
- typescript
Try it out
https://astonhack2021.vercel.app/
https://astonhack2021-bnct2bqmk-asobirov.vercel.app/
https://github.com/asobirov/astonhack2021
Created By
Akbarshokh Sobirov
Milosz Paszkowski
Michael Monfries