Face Detection and Recognition in C# using EmguCV 3.0 (OpenCV Wrapper) – Part 1

Introduction

First off, Face detection and Face recognition are two totally different things although one builds upon the other (recognition builds upon detection). Detection is the process by which the system identifies human faces in digital images, regardless of the source while Recognition is the identifying a known face with a known name in digital images, still regardless of the source. The source can range from a scanned copy of a photograph to a live video stream. Face detection and recognition is a section of Machine learning with a good number of research topics focused on improving the existing algorithms.
This article intends to show the reader how to use EmguCV 3.0 for Face detection and recognition in C#, emphasis on 3.0 because a lot of changes have been made to the library since 2.x versions, and a lot of tutorials/articles (as at the time of writing) focus on the 2.x versions of the library.

What is EmguCV?

Emgu CV is a cross platform .Net wrapper to the OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++ etc. The wrapper can be compiled by Visual Studio, Xamarin Studio and Unity, it can run on Windows, Linux, Mac OS X, iOS, Android and Windows Phone. Source: EmguCV Home
In plain English, EmguCV allows you to use OpenCV (a C++ image library) on the .net platform with languages like C#.

Pre-Requisites, Before Continuing

The scope of this article doesn’t cover how to setup EmguCV but before continuing with the rest of this article, you would need to setup EmguCV 3.0. To do so, kindly follow the guide in the official link below:
Setting Up EmguCV
Also, this tutorial on CodeProject (Creating Your First EMGU Image Processing Project) captures how to set EmguCV up for development. The steps (for set-up) shown in the tutorial applies for version 3.0 too.

This article splits the process of detection and recognition into the following steps:

  1. Getting the Camera to work i.e to provide live feed
  2. Detecting human faces from the feed provided by the camera
  3. Saving the detected face with an identifier
  4. Training our recognizer with the saved faces
  5. Carrying out recognition against the trained system

Implementing a Working Camera

To get live feed from our installed camera, we would need to import EmguCV UI tools into our visual studio. To do that, lets follow the offical guide: Add ImageBox Control.
Drag the control onto the form and rename it to imgCamUser, you can re-size as you please.
After that has been added successfully, we use the Capture class to get the live feed from our Camera.

 private Capture _capture;

 _capture = new Capture();

imgCamUser.Image = _capture.QueryFrame();

With the above code implemented, we should be seeing live feed from the computer’s default camera show in our ImageBox control.

Implementing Face Detection

Now, we have our live feed, we need to be able to detect face(s) from the feed.

Popularly used for face detection is the Viola-Jones algorithm because it is the easiest ready to use face detection method which is supported by EmguCV and has proven to return great results. In using this algorithm, we would be making use of a face detector called Cascade Classifier/Detector that has been trained on thousands and thousands of human faces (remember face detection is a subject under machine learning). The training data generated from the faces are stored in xml files which come by default with the EmguCV package which are stored in [EnguCV Root Folder]\opencv\data\haarcascades
For the sake of this article, we choose to use the haarcascade_frontalface_alt_tree.xml. I sampled the other xml files, but this gave me my best result yet. There is no harm in trying the others, they might work well for you.
I moved the haarcascade_frontalface_alt_tree.xml file to my application folder and i set the property ‘Copy to Output Directory’ to ‘Always’.
Actual implementation of the detection goes forth:

 private CascadeClassifier _cascadeClassifier;
_cascadeClassifier = new CascadeClassifier(Application.StartupPath + "/haarcascade_frontalface_alt_tree.xml");
using (var imageFrame = _capture.QueryFrame().ToImage<Bgr, Byte>())
                {
                    if (imageFrame != null)
                    {
                        var grayframe = imageFrame.Convert<Gray, byte>();
                        var faces = _cascadeClassifier.DetectMultiScale(grayframe, 1.1, 10, Size.Empty); //the actual face detection happens here
                        foreach (var face in faces)
                        {
                            imageFrame.Draw(face, new Bgr(Color.BurlyWood), 3); //the detected face(s) is highlighted here using a box that is drawn around it/them
                           
                        }
                    }
                    imgCamUser.Image = imageFrame;                    
                }

We should notice that our classifier/detector’s DetectMultiScale method takes four(4) parameters. I would breifly explain what the parameters are:
The first parameter is the grayscale image i.e the actual image we want to detect the face from. Notice it has been converted to a gray frame using the Convert in the code snippet above.
The second parameter is the scale factor. This parameter must be greater than 1.0 and the closer it is to 1.0 the longer it will take to detect faces but there’s a greater chance that you will find all the faces. I don’t mind my app taking its time to give me adequate result.
The third parameter is the minimum number of nearest neighbors. The higher this number the fewer false positives you will get.
The last parameter are the max size in pixels. Notice i set to empty.
Please keep in mind that the other three (3) parameters are optional, while the only required parameter for the method is the Image.

That’s about it for face detection. In the second part of this article, i would discuss and show how face recognition can be implemented. How you can train your recognition engine and store training data, and how based on the training data, you can accurately predict (or at least try to) which face is supplied.
Also as an added bonus, we would be storing our images and related data in a structured database using SQLite.

Enjoy!

Part 2 – Read here

  • Erik Sandoval

    Thx a lot for sharing Ahmed, I will look forward for you next post about face recognition….Here is my test page with the results: http://www.edatasoluciones.com/FaceDetection/FaceLogin

    • Ahmed

      Hi Erik,
      Thanks for reading.
      I should post that in a few days. Would be sure to notify you.
      And great results by the way!
      Ahmed.

      • Erik Sandoval

        Well, I´m looking for info on Internet about Face Recognition and your post is the right solution for me so far.Basically Im trying to develop a webpage where students could get personalised information using face recognition as the entry method…. Thx a lot!

    • Ahmed
      • Erik Sandoval

        OMG!!! You´re my hero Ahmed, I will try to use it on to next weeks and I will show you the results.

      • Erik Sandoval

        Hello again Ahmed, I´m still developing the recognition features; found a way to read data from any image url it work for public webcam too so im testing it on my personal ip camera to recognize people too 🙂
        I want to share my progress with you. thanks in advance for your support

        http://natvelfashion.com/test/DownloadComCam.aspx

        • Ahmed

          Awesome stuff eric, Seems like you are onto something quite related to Policing/Security related.
          Always happy to help.
          Cheers!

          • Erik Sandoval

            I want to help with security at my community/city cause things are really bad here. I will be in touch as soon I get more progress

            If you wish to join Im opening a Facebook group trying to make this stuff to work 🙂 Regards!!

            https://www.facebook.com/Community-Cam-795415473877593/

  • Bora B.

    Great article, thanks. Looking forward to the facial recognition part!

  • Erik Sandoval

    Hello Ahmed, I did a workaround reading the images from database and saving them to a local folder, then reading the images in the folder for Face Detection. Im still searching to read the image from a direct stream created from the file in the database if its possible. My test page: http://www.edatasoluciones.com/FaceDetection/FaceDataBase

    • Erik Sandoval

      Finally made it!! just one more day of coding helped me to got the workaround.. I used an image handler to read the image from database (ashx) and then send the url of image hanlder to the HaarCascade. Yahooo!

      • Ahmed

        Hi Erik,
        Good to see you are making progress on your project.
        However, have you considered the resources and response time involved in reading and building the image from the database?
        Cheers!

  • ADRIÀ MANZANO JUNCOSA

    Hi Ahmed, you will post the 2 part tutorial?. Thanks in advance ^^

  • Roy Ritche Vy

    Hi I used your code and I’m getting an error in the detecmultiscale part. Apparently, the DetectMultiScale needed 5 parameters to work so I used two size.empty. When I did, it displayed a “NullReferenceException was unhandled” error. any kind of help would be appreciated. :))

    • Ahmed

      Hi Roy,

      By default, the only required parameter for the DetectMultiScale method of the CascadeClassifier is the IInputArray which is the source image. Other parameters/arguments are optional as they have a default value. If you are getting a null exception, i would advice you step through your code and inspect the image before it is passed to that method to make sure you aren’t passing null.

      • Roy Ritche Vy

        Thanks for the answer Ahmed. So I was tinkering with my code a bit and now it looks alot like your code. But I am getting an error in this line
        using (var imageFrame = capture.QueryFrame().ToImage())
        particularly on “ToImage”. The debugger is saying that I am missing a directive or an assembly reference. I just want to ask which namespace ToImage is from? I think that may be the root of my problem.

  • Stijn Pillaert

    The opencv_highgui220.dll file is missing in my ‘bin’ folder so I can’t add it to my project and I keep getting this error:
    Unable to load DLL ‘opencv_highgui220’: The specified module could not be found. (Exception from HRESULT: 0x8007007E)

    I downloaded it again but the opencv_highgui220 is still missing. Any solution?
    (EmguCV download link: http://sourceforge.net/projects/emgucv/)

    • Hakeem Ojulari

      Current version of OpenCV doesn’t have that anymore. Try to download OpenCV 1.0 or 2.0. Thanks

  • Usman Saeed

    AslamoAlaikum
    How are you Ahmad
    Can you help me out with face detection/recognition problem.
    I am developing Auto Attendance System using Live Camera View i.e., Video Face Recognition
    Below is my code , I am stuck in detection process and getting null value exception error and I mentioned it as comment.
    Please help me with this I will be very thankful to you.

    using System.ComponentModel;

    using System.Data;

    using System.Drawing;

    using System.Linq;

    using System.Text;

    using System.Windows.Forms;

    using Emgu.CV;

    using Emgu.CV.Structure;

    using Emgu.Util;

    using Emgu.CV.CvEnum;

    namespace Haderoon

    {

    public partial class MainForm : Form

    {

    // global variables

    private Capture capture;

    private bool captureInProgress;

    private HaarCascade haar;

    public MainForm()

    {

    InitializeComponent();

    }

    private void processFrame(object sender , EventArgs args)

    {

    Image ImageFrame = capture.QueryFrame();

    if (ImageFrame != null)

    {

    Image grayframe = ImageFrame.Convert();

    var faces = grayframe.DetectHaarCascade(haar, 1.4, 4,

    HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,

    new Size(25, 25))[0]; // Here I am getting Null Value

    foreach (var face in faces)

    ImageFrame.Draw(face.rect, new Bgr(Color.Green), 3);

    }

    liveBox.Image = ImageFrame; //liveBox is an EmguCV imageBox

    }

    private void btnStart_Click(object sender, EventArgs e)

    {

    #region if capture is not created, create it now

    if (capture == null)

    {

    try

    {

    capture = new Capture();

    }

    catch (NullReferenceException excpt)

    {

    MessageBox.Show(excpt.Message);

    }

    }

    #endregion

    if (capture != null)

    {

    if (captureInProgress)

    {

    btnStart.Text = “Start”;

    Application.Idle -= processFrame;

    }

    else

    {

    btnStart.Text = “Stop”;

    Application.Idle += processFrame;

    }

    captureInProgress = !captureInProgress;

    }

    }

    private void MainForm_Load(object sender, EventArgs e)

    {

    haar = new HaarCascade(“haarcascade_frontalface_default.xml”);

    }

    private void releaseData()

    {

    if (capture != null)

    capture.Dispose();

    }

    }

    • MOeez Raja

      SALAM Usman
      have over come this problem same exception is come in my program and i didn’t find any solution to solve it. If you have please share it to me.

      • Hakeem Ojulari

        The above code was used in EmguCV 2.0 downward I guess. HarrCascade was not defined in 3.xx

  • Faisal M

    Ahmad, I have a requirement where we will have to blur all faces present in an uploaded video due to privacy concerns. Videos can range in length from 5 mins to 60 mins long.
    So my question is, using this lib of yours is it possible to do things with good performance especially speed wise?
    If not then, is there any other good n free alternative available in C# to do this thing?

    • Ahmed

      Hi,

      Three ways that i can think of off the top of my head:

      1. Draw a rectangle with a high level of thickness . This would cover the detected face with the drawn rectangle. See this line: imageFrame.Draw(face, new Bgr(Color.BurlyWood), 3). This works but isnt really “clean”

      2. Use the EmguCVs Image.SmoothBlur and pass the width and height of the detected face (it is a rectangle) to the method. But this is a guess. There isn’t any way to specify the x and y of the rectangle

      3. Use http://notes.ericwillis.com/2009/10/blur-an-image-with-csharp/ pass the Image and the face (which is a rectangle) and returns the blurred image and you can display that. What you would need to take care of is to apply the blur before you display the image in picture/image box. i.e. immediately after you query your capture source.

  • Erik Sandoval

    Ahmed has been a long time. Once again I want to thank you for your post on how to detect and recognize faces which made ​​me understand a little more about how the face detection mechanisms work. Right now I ‘m working on some tests using Twitter as a data source for faces: http://www.edatasoluciones.com/FaceDetection/Engine/TwitterDetection

    • Ahmed

      Excited to know you are making progress! Feel free to share whatever you accomplish or learn.
      Cheers!

      • Erik Sandoval

        Ahmed, I Finally got a database, just for fun, with +100,000 faces detected from social media posts that keeps growing minute after minute with social media analysis. I want to share with you a page where you can see the results live: http://www.edatasoluciones.com/FaceDetection/Engine/Facewall

      • Erik Sandoval

        Hi Ahmed I finally accomplished to get a database of +100,00 faces from the previous process. Probably I will try the recognition feature as the next step. By the way I changed a little bit the interface so you can check the last faces captured live on the next url: http://www.edatasoluciones.com/FaceDetection/Engine/FaceWall

  • MOeez Raja

    Salam Ahmed.

    Nice work
    i have one question to you can we done the facial detection and recognition in the WPF application. Like I have seen all the examples made in the C# are made in the Visual studio FORM application.
    best regards MOeez

  • Ahsan

    Ahmed what is this FrmSaveDialogue ();
    plz give details .. couldnt understand
    my visual studios 2015 is giving errors

  • Phoibos

    Ahmed help me please. how can i make it accurate recognize face. Example If face is not save at local database i want to show unknown face.. I use local database to save faces…. this is my source https://www.codeproject.com/Articles/239849/Multiple-face-detection-and-recognition-in-real.

  • Tanzeel U Rehman

    Ahmed Bro JzakAllah for this infromation. I have a question about EMGU CV that either it supports Real time Face+Eye Detection and recognition or not ? and if yes then please guide me i.e. Road map to learn this?

  • emmanuel ortiz

    Hey bro good day could you help me with some guideness with the face recognition Emgu CV with C#.
    I could not find your email in this web page, could you send me one?

    ortizemmanuel219@gmail.com

  • Binod Bomzzn

    Hello,Could you send me sample of the project. Thanks.
    Bomzznbinod@gmail.com

  • Show How

    Hi everybody, If you are looking to get step by step face detection ans recognition on your visual studio for real time, just take a look at my fiverr, don’t hesitate to contact me
    https://goo.gl/BDiK4i

  • Tushar Singh

    hello, i’m making a project on face recognition and detection. So could you please give me the tutorial. I am running short of time and make it soon as possible. please give me necessary tutorial.

  • John Arthur

    Hey bro tanzel, can you send me codes for face and eye detection in emgucv?
    Here’s my email : arthurwongchuyy@gmail.com

  • Abid ali

    if someone need face detection software with attendance then contact me through my email

    abidali313peace@gmail.com

  • Erik Sandoval

    Whatt do you guys thnik about this project? http://www.edatasoluciones.com/SecurityDetection/Engine/RecognitionEngine_V2_0
    It keeps track of posts on Social Media as first step to detect human faces. As a second step its to apply the recognition process. Any idea if it could be applied to a related business process inside an organization?

  • evrey matem

    Hello, Ahmed! So haarcascade_frontalface_alt_tree.xml doesn’t detect face. but default is detecting face. I copy your code. But it doesn’t work. I’m using EmguCV version 3.1. Thanks for answer. Can you help me?

  • أحمد صوالحة

    Dir sir, Thanks for this great post, but I emplented the code , It works fine, but how to detecte not found faces?? If I check for a face not in the database, it selects one of the existing faces, It never says ( not found ) .. How to do that please?
    I read this article and can’t do anything:
    https://www.codeproject.com/articles/261550/emgu-multiple-face-recognition-using-pca-and-paral

    I converted the code to VB.net
    thanks

  • Tanzeel U Rehman

    Ahmed bro can i produce a beep if face is not detected?