Türk İşaret Dili Çevirisi
A real-time recognition system that translates camera input into the Turkish Sign Language finger alphabet.
- Role
- Timeline
- Stack
Problem
Even the simplest exchange between someone who signs and someone who does not requires an interpreter. Ready-made solutions for Turkish either do not exist or are built on the English alphabet; Turkish's own letters (ç, ğ, ı, ö, ş, ü) appear in none of them.
Goal
Build a model that recognises letters with an ordinary laptop camera, no extra hardware, with no perceptible lag.
Architecture
- 01
Capture
Frame stream from the camera, load balanced with frame skipping.
- OpenCV
- VideoCapture
- 02
Preprocess
Cropping the hand region, normalisation, resizing.
- ROI
- Normalize
- Resize
- 03
Model
Convolutional neural network; letter classification.
- CNN
- TensorFlow
- Keras
- 04
Postprocess
Smoothing per-frame predictions over a time window.
- Smoothing
- Threshold
- 05
Output
Appending the stabilised letter to the text buffer.
- Text buffer
Technical decisions
- ChoseTensorFlow / KerasOverPyTorch
I was changing the architecture several times a day. Keras's short loop helped more at this stage than PyTorch's flexibility.
- ChoseCropping the hand regionOverFeeding the raw frame
With raw frames the model was learning the background — the same sign in a different room went unrecognised. Cropping fixed accuracy substantially without growing the dataset.
- ChoseTime-window smoothingOverPer-frame prediction
Per-frame prediction made the output flicker; nonsense letters appeared while the hand moved. Taking the majority of the last N frames made the screen readable.
Challenges
Letters that look alike
In the finger alphabet some letters differ only by thumb position, and the model kept confusing them. I looked at the confusion matrix and collected targeted data only for those classes — growing the right part beat growing the whole dataset.
from collections import deque, Counter
class LetterStabilizer:
"""Kare başına tahminleri pencereler, kararlı harfi döndürür."""
def __init__(self, window: int = 12, min_ratio: float = 0.7):
self.buffer = deque(maxlen=window)
self.min_ratio = min_ratio
def push(self, letter: str, confidence: float) -> str | None:
if confidence < 0.6:
return None
self.buffer.append(letter)
if len(self.buffer) < self.buffer.maxlen:
return None
candidate, count = Counter(self.buffer).most_common(1)[0]
if count / len(self.buffer) >= self.min_ratio:
self.buffer.clear()
return candidate
return NoneStaying real-time
Running the model on every frame dropped the framerate on a laptop. By skipping frames and running inference on a separate thread I decoupled the video stream from the model; the camera stays smooth while prediction proceeds at its own pace.
By the numbers
- 0
- Letters in the alphabet
- 0
- Frame smoothing window
- 0
- Extra hardware required
Takeaways
The fastest way to understand a model's problem is the confusion matrix. "Accuracy is X percent" says nothing; which class is confused with which says everything.
Collecting data took much longer than tuning the model, and affected accuracy far more.