Sign Language Detection Using open-cv and Haar cascade

 Sign Language Detection Using open-cv python

Haar Cascade classifiers are an effective way for object detection. This method was proposed by Paul Viola and Michael Jones in their paper Rapid Object Detection using a Boosted Cascade of Simple Features .

The algorithm implemented in OpenCV can also be used to detect other things, as long as you have the right classifiers. My OpenCV distribution came with classifiers for eyes, upper body, hands, frontal face and profile face.

Haar Cascade is a machine learning-based approach where a lot of positive and negative images are used to train the classifier.

Frequently used keywords

haar

Positive Images: These images contain the images which we want our classifier to identify.

Negative Images: Images of everything else, which do not contain the object we want to detect.

Features

•  Initially, the algorithm needs a lot of positive images (images of hand) and negative images (images without hand) to train the classifier. Then we need to extract features from it. For this, Haar features shown in the below image are used.
• Each feature is a single value obtained by subtracting sum of pixels under the white rectangle from sum of pixels under the black rectangle.

Now, all possible sizes and locations of each kernel are used to calculate lots of features. (Just imagine how much computation it needs? Even a 24x24 window results over 160000 features). For each feature calculation, we need to find the sum of the pixels under white and black rectangles. To solve this, they introduced the integral image. However large your image, it reduces the calculations for a given pixel to an operation involving just four pixels. Nice, isn't it? It makes things super-fast.

The final classifier is a weighted sum of these weak classifiers. It is called weak because it alone can't classify the image, but together with others forms a strong classifier.

The paper says even 200 features provide detection with 95% accuracy. Their final setup had around 6000 features.

Detecting the object in an image

So now you take an image. Take each 24x24 window. Apply 6000 features to it. Check if it is hand or not. Wow.. Isn't it a little inefficient and time consuming? Yes, it is. The authors have a good solution for that.

In an image, most of the image is non-hand region. So it is a better idea to have a simple method to check if a window is not a hand region. If it is not, discard it in a single shot, and don't process it again. Instead, focus on regions where there can be a hand. This way, we spend more time checking possible hand regions.

For this they introduced the concept of Cascade of Classifiers. Instead of applying all 6000 features on a window, the features are grouped into different stages of classifiers and applied one-by-one. (Normally the first few stages will contain very many fewer features). If a window fails the first stage, discard it. We don't consider the remaining features on it. If it passes, apply the second stage of features and continue the process. The window which passes all stages is a hand region. How is that plan!

visualization of open-cv face detection

Training the Cascade Classifier:

Step1: To gather a bunch of positive(the images that we want to train) and negative(images that do not contains the object we want to train) images.

Step2: To get the hand.info file which will require to train the positive images.

Step3: To describe all the negative images we simply collect their names in bg.txt file.

Step4:  We now create a vec file using OpenCV. In the command prompt(in the root folder) run the command

opencv_createsamples -info hand.info -num 550 -w 48 -h 24 –vec hand.vec

Step5:  In order to train cascade, we will now create a directory named data and run

opencv_traincascade -data data -vec hand.vec -bg bg.txt -numPos 500 -numNeg 500 -numStages 10 -w 48 -h 24 -featureType LBP

Step6: Now in our data folder, we have cascade.xml , which is our final cascade and can be used for hand detection. It also contains stage wise xml files after each stage.

How to use the classifier

• Firstly we will use the CascadeClassifier function of OpenCV to point to the location where we have stored the XML file    

  hand = cv2.CascadeClassifier(‘haarcascade/hand.xml’)

• we will now try to locate the exact features in our hand. By using the following function                                                     handDetect = hand.detectMultiScale(img, 1.3, 5)

• From the above step, the function detectMultiScale returns 4 values — x-coordinate, y-coordinate, width(w) and height(h) of the detected feature of the hand.
• We will use this four values to draw a rectangle around the hand or we can put text on the screen as follows 

for (x, y, w, h) in handDetect:
        cv2.putText(frame, 'Hand', (80, 50), cv2.FONT_HERSHEY_SIMPLEX, 2, (0, 255, 255), 2, cv2.LINE_AA)

Link for the technical documentation:

                            Document

Here is an Demo of the Project