Face Detection and Recognition in C# using EmguCV 3.0 (OpenCV Wrapper) – Part 2

A quick recap

In the first part of this article, we saw how we can initialize our default camera, get image frames from it and carry out face detection.
To complete the scope of this article, we would be taking a step further. We would look at how to store training data in a structured database, train your face recognition engine and use to engine to predict faces detected in images.

Store Training Data (Using SQLite)

The storage of training data would be demonstrated using SQLite due to the fact that it is easy to use. SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. It removes the hassle of client-server communication when it comes to using databases.

Although it should be pointed out that this procedure can work for all types of database.
To create and manage my SQLite database, i make use of Navicat Premium although there are other free alternatives located at this page – SQLite Management Tools, feel free to use any of them.
The structure of the database used in this article is pretty simple, it contains one user-defined table which is named faces and a system generated table called sqlite_sequence which SQLite engine uses to keep track of the table IDs.
The faces tables holds the training data from the detected images and the corresponding user details associated with them. Below is a screen shot of the table structure.

Faces Table

Faces Table – Face Recognition Tutorial


From the image above, you can see four table fields.

  1. id: Primary Key, keeps all the records uniquely identified and it is auto-incremented starting from 1
  2. username: A friendly name to associated with the face.
  3. faceSample: The detected and captured face is converted from Image < Gray, byte > format to a byte[] format and stored in this field. Notice the field is declared a BLOB data type.
  4. userId: This contains integer values identifying the face. It works like the username, but the recognition engine in EmguCV associates numbers (as labels) with Image rather than strings, hence we need Integer identifiers for the faces for the engine, and username identifier for the users to understand. If this isn’t clear yet, we would see in a minute.

Having setup our database structure as we want it, we can proceed to code away!

Storing the recognized face

Foremost, you need the ADO.NET Data Provider for SQLite before you can communicate to your SQLite database in C# so you would have to add the System.Data.SQLite reference.
While this article isn’t focusing on the database operations, i would just want to take a minute to discuss how the storage of the recognized face data and other associating details is handled. For simplicity sake, i created an interface that has the basic methods i need. Here is the interface declaration.

interface IDataStoreAccess
    {
        String SaveFace(String username, Byte[] faceBlob);

        List<Face> CallFaces(String username);

        bool IsUsernameValid(String username);

        String SaveAdmin(String username, String password);

        bool DeleteUser(String username);

        int GetUserId(String username);

        int GenerateUserId();

        String GetUsername(int userId);

        List<String> GetAllUsernames();
    }

The methods defined in the interface are implemented in a separate class. The methods handle the CRUD (Create, Read, Update and Delete) operations that involve our database. From the method names, the functions are quite understandable.
I also have a Face class defined that represents a face object. When i need to select faces from my database, the CallFaces method defined in the interface class returns a list of face objects that i can easily work with. See the attached (at the end of this post) implemented class named DataStoreAccess.cs for details on how i implemented the methods in the interface classes.

internal class Face
    {
        public byte[] Image { get; set; }

        public int Id { get; set; }

        public String Label { get; set; }

        public int UserId { get; set; }
    }

Now, the question is how do we get the detected face passed to the SaveFace method in our data access class. Below shows the code that does that. The code below also handles the conversion of the captured/detected face to a byte[] format suitable for our defined table structure.

var faceToSave = new Image<Gray, byte>(picCapturedUser.Image.Bitmap);
            Byte[] file;
            IDataStoreAccess dataStore = new DataStoreAccess(_databasePath);
            var frmSaveDialog = new FrmSaveDialog();
            if (frmSaveDialog.ShowDialog() == DialogResult.OK)
            {
                if (frmSaveDialog._identificationNumber.Trim() != String.Empty)
                {
                    var username = frmSaveDialog._identificationNumber.Trim().ToLower();
                    var filePath = Application.StartupPath + String.Format("/{0}.bmp", username);
                    faceToSave.ToBitmap().Save(filePath);
                    using (var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read))
                    {
                        using (var reader = new BinaryReader(stream))
                        {
                            file = reader.ReadBytes((int) stream.Length);
                        }
                    }
                    var result = dataStore.SaveFace(username, file);
                    MessageBox.Show(result, "Save Result", MessageBoxButtons.OK);
                }

            }

The method above can be attached to a user triggered event e.g. After a face is detected, the user clicks a save button that calls this method. It can also be automatically called when a face detection happens. The code above also displays a dialog box prompting the user to supply a username for the face that is about to be saved.

>>>>See previous post on how to do face detection.<<<

To make the training and recognition engine work better, we need to capture multiple faces of the same user. Our username is not a key in our database table, so we can have multiple records with the same username. Having multiple faces of a user helps the engine get trained better. Like i mentioned in the previous post, Face recognition is field in Machine learning and as we all know, machine learning greatly relies on a lot of existing data in order to make accurate/near accurate predictions in the future.

Training the recognition engine

After having saved our data, we need to train our recognition engine with the dataset we have available. The time at which this takes to complete a training sequence is determined by how large your dataset is. Below shows the code that trains our recognizer engine.

public bool TrainRecognizer()
        {
            var allFaces = _dataStoreAccess.CallFaces("ALL_USERS");
            if (allFaces != null)
            {
                var faceImages = new Image<Gray, byte>[allFaces.Count];
                var faceLabels = new int[allFaces.Count];
                for (int i = 0; i < allFaces.Count; i++)
                {
                    Stream stream = new MemoryStream();
                    stream.Write(allFaces[i].Image, 0, allFaces[i].Image.Length);
                    var faceImage = new Image<Gray, byte>(new Bitmap(stream));
                    faceImages[i] = faceImage.Resize(100,100,Inter.Cubic);
                    faceLabels[i] = allFaces[i].UserId;
                }
                _faceRecognizer.Train(faceImages, faceLabels);
                _faceRecognizer.Save(_recognizerFilePath);
            }
            return true;
           
        }

What we did up there is pretty straightforward. We declared a new instance of EigenFaceRecognizer which is an EmguCV class as _faceRecognizer, then we proceeded to call all the faces we have stored in our database as List< Face >. Each of the faces were converted to Image < Gray, byte > and resized and passed to the Train method of the EigenFaceRecognizer class. The Train method takes two main parameters; the first is the Array of Images, and the second is the Array of Integer Labels of the images. The label is what is returned when the face is recognized, which we would see how that is done later.
After training our engine, we need to save the current state of the model, in order to carry out recognition. EmguCV allows the model state to be saved as a YAML file. You can carry out more training as soon as you have more faces in your data store.

Recognizing Faces

So far, we have been able to store our training data (detected faces), trained our recognizer engine, store our engine model state. We move on to the best part – Recognizing the face.
We use the Predict method of the recognizer engine to recognize the specified face. The method takes just one parameter which is the Image of the face we are trying to recognize or predict. Here, lets see:

_faceRecognizer.Load(_recognizerFilePath);
public int RecognizeUser(Image<Gray, byte> userImage)
        {
            _faceRecognizer.Load(_recognizerFilePath);
            var result = _faceRecognizer.Predict(userImage.Resize(100, 100, Inter.Cubic));
            return result.Label;
        }

0 is returned if the face cannot be predicted, and the Integer Label is returned if it was predicted successfully. This integer label returned matches the userId field in our database table. We can use this integer label returned to fetch the user details from the database. The method to do this is also in the data store access interface shown earlier.

Resources

In case you get stuck, here are the major classes you can look through to understand what was described in this article. And you can always use the comment section to make clarifications.
DataStoreAccess.cs
Face.cs
IDataStoreAccess.cs
RecognizerEngine.cs

Thanks for reading!

  • Pingback: » Face Detection and Recognition in C# using EmguCV 3.0 (OpenCV Wrapper) – Part 1 - Ahmed's Blog()

  • Turgay Türk

    Nice article Ahmed. Please check the download links. Seems you have mixed up the file names! RecognizerEngine.cs directs to Face.cs.

    • Ahmed

      Ooops! Thanks for pointing that out. Fixed!

      • shrikant yadav

        Sir how can we detect whether a face is streaming live or its an photograph? Something like EyesBlink detection.

        • Ahmed

          Hi,

          To achieve this, you would need to implement some form of motion detection as Eyes blinking can be detected as a form of “movement”. Luckily OpenCV can be used for motion detection too.

  • BR

    In my dataset over 10000 images and I can not train my recognition engine because of
    out of memory error. How to resolve this problem? Thanks.

  • BANDHAN MUNJAL

    emgufacerecognizer class is not there in emgu.cv.dll version 3.0

    • BR

      using Emgu.CV.Face

      • Asad Abbas

        I am facing one problem in your given soucre code.
        The solution is not build because it gives the following error.
        “This project references NuGet package(s) that are missing on this computer.”

  • lac doan ba son

    Nice article Ahmed. Where do i download the source code. And i want to save trained face to sql server, how to do this.

  • Rnc

    nice article, this tutorial can do with monodevelop on raspberry pi?

    • Ahmed

      I haven’t tried this before, but should be possible.

  • Manikanta Santhosh Kuchimanchi

    thank you for your detailed information :). I tried registering only one face. and after everybody comes infront of camera, it is returning the same id.

    • Ahmed

      thats a bit strange, but for your recognizer to work well, try registering multiple faces of the same user so your engine can be trained well.

    • Rezell

      Hi, you help me? I’m having the same problem, what did you do to fix it? Thank you so much!

  • kapila Rathnayaka

    how to convert the detected faces to bitmap for saving and recognizing ?

    • KotZelenkin

      //-> Our image
      Image image;
      //-> Convert it to Bitmap
      Bitmap crop_result = image.ToBitmap();
      //-> Convert it to shades of gray
      Image grayImage = image.Convert();
      //-> To skip errors we use “try{}… catch{}”
      try
      {
      //-> Train detector
      var Faces = Cascade.DetectMultiScale(grayImage, 1.3, 5);
      // Processing each face we found
      foreach (var face in Faces)
      {
      Rectangle face_rect = new Rectangle(face.X, face.Y, face.Width,face.Height);
      //-> Here you need to crop the “face_rect” from “image”
      crop_result = image.ToBitmap().Clone(face_rect, bmpImage.PixelFormat);
      // Here you can save “crop_result”
      // * * *
      //<-
      }
      //<-
      }
      catch { }

    • KotZelenkin

      .csharpcode, .csharpcode pre
      {
      font-size: small;
      color: black;
      font-family: Consolas, “Courier New”, Courier, Monospace;
      background-color: #ffffff;
      /*white-space: pre;*/
      }

      .csharpcode pre { margin: 0em; }

      .csharpcode .rem { color: #008000; }

      .csharpcode .kwrd { color: #0000ff; }

      .csharpcode .str { color: #a31515; }

      .csharpcode .op { color: #0000c0; }

      .csharpcode .preproc { color: #cc6633; }

      .csharpcode .asp { background-color: #ffff00; }

      .csharpcode .html { color: #800000; }

      .csharpcode .attr { color: #ff0000; }

      .csharpcode .alt
      {
      background-color: #f4f4f4;
      width: 100%;
      margin: 0em;
      }

      .csharpcode .lnum { color: #606060; }

      1: //-> Our image
      2: Image<Bgr, Byte> image;
      3: //-> Convert it to Bitmap
      4: Bitmap crop_result = image.ToBitmap();
      5: //-> Convert it to shades of gray
      6: Image<Gray, Byte> grayImage = image.Convert<Gray, Byte>();
      7: //-> To skip errors we use “try{}… catch{}”
      8: try
      9: {
      10: //-> Train detector
      11: var Faces = Cascade.DetectMultiScale(grayImage, 1.3, 5);
      12: //<-
      13:
      14: //-> Processing each face we found
      15: foreach (var face in Faces)
      16: {
      17: Rectangle face_rect = new Rectangle(face.X, face.Y, face.Width,face.Height);
      18: //-> Here you need to crop the “face_rect” from “image”
      19: crop_result = image.ToBitmap().Clone(face_rect, bmpImage.PixelFormat);
      20: //<-
      21: //-> Here you can save “crop_result”
      22: // * * *
      23: //<-
      24: }
      25: //<-
      26: }
      27: catch { }

    • KotZelenkin

      Here is cropping and conversion to bitmap. I’m just working on this.

    • KotZelenkin

      Here is cropping and conversion to bitmap. I’m just working on this.

  • Usman Rasheed

    Ahmed brother i implemented facial recognition by using this code it works fine but the problem is that it recognize even those face those are not in trained file it gives a result every time how can i know if any face is not in trained file.

    • zahid

      It depends upon the training data set, the algorithm used for recognition and parameters for recognition. Try to train it with more known face. Try to use another algorithm and find the best among them. You can use another algorithm in Constructor of RecognizerEngine class as:
      _faceRecognizer = new EigenFaceRecognizer(80, double.PositiveInfinity);
      OR
      _faceRecognizer = new FisherFaceRecognizer(80, double.PositiveInfinity);
      OR
      _faceRecognizer = new LBPHFaceRecognizer(1, 8, 8, 8, double.PositiveInfinity);
      Moreover try to find the best parameter as per your need.

  • Haris Ali

    I cant fix this error when i open the source code u provided in the visual studio. please help

    Error 2 This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is ..packagesSystem.Data.SQLite.Core.1.0.97.0buildnet40System.Data.SQLite.Core.targets. C:UsersHomeDocumentsVisual Studio 2013ProjectsProject_FaceRecognitionProject_FaceRecognition.csproj 261 5 Project_FaceRecognition

    • zahid

      In Package Manager Console, Type Install-Package System.Data.SQLite.Core

  • Sonia

    Hello Ahmed
    I tried using your code for face recognition. But I don’t know why FRmMain page ofyour source code is not opening
    May be it has problem with this line
    “private readonly String _trainerDataPath = Application.StartupPath + “/traineddata”;”
    as i don’t know what does trained data contains.
    Can you pleasetell me why it is not working

    • Ahmed

      Hi Sonia,
      So the training results are saved in that file. The file is created by the RecognizerEngine when training is being done. That shouldn’t be a problem. Perhaps you can paste the exact error you are having here?

  • Ajit

    Hi Ahmed,
    I am using emgucv Version 3.1.0.2282. I am having compile issues in the class RecognizerEngine.

    Issue 1: In the method TrainRecognizer(), wWhile calling the Train method: _faceRecognizer.Train(faceImages, faceLabels);
    Argument 1: cannot convert from ‘Emgu.CV.Image[]’ to ‘Emgu.CV.IInputArray’
    Argument 2: cannot convert from ‘int[]’ to ‘Emgu.CV.IInputArray’

    Issue 2:
    On the method RecognizeUser(Image userImage):
    Argument 1: cannot convert from ‘Emgu.CV.Image’ to ‘Emgu.CV.IInputArray’

    Pardon me if I am missing something silly, I am a newbie and this is my third program with emgucv. I would appreciate your assistance.

  • Ajit

    I was able to make it work successfully on Emgucv v 3.1.0.2282. Very well done Ahmed and thanks for posting the code.

    • Oussama Bensalem

      I hope my message finds you in peace. I used as you Emgucv v 3.1.0.2282 but with LPBHFaceRecognizer because the EigenFaceRecognizer did not work.
      But Even with LPBHFaceRecognizer I’m always geting 0 return I tied every thnig but it did not work need your help

    • noman abbasi

      Hey bro ..can u send me the code of face detection and recognition complete ..it’s my final year project ..help me plz

  • Pauline

    From where can i get Emgu.CV.FaceRecognizer library please?

    • Ajit

      Pauline: Are you using the code shared (below) by Ahmed as a base: https://drive.google.com/file/d/0B3y5W7yHmJh-VkRHUGhnUEQ3QjQ/view?usp=sharing
      The project has a file named FaceRecognizer.cs. Let me know if you have any other questions. I would be glad to help.

      • Pauline

        So, I just import that class and make reference to it? I have been using EigenObjectRecognizer. Do not have any results so far 🙁

        • Ajit

          That might solve the problem. I will share the screenshots of my solution explorer and other project settings in a day or two.
          I will also try to build the solution from scratch and document the steps. This might take about a week.

        • Ajit

          Here are the screenshots:

  • Asad Abbas

    Thanks Ahmed for sharing.
    But I am facing one problem in your given soucre code.
    The solution is not build because it gives the following error.
    “This project references NuGet package(s) that are missing on this computer.”

    • Ajit

      As Zahid mentioned below:
      In Package Manager Console, Type Install-Package System.Data.SQLite.Core

      • MOeez Raja

        Ajit I have follow your instruction but is not working the same error is comming again and again

        • Ajit

          Hi Moeez, It has been a while since I worked with the project. I will try to set it up on a new machine and record the steps.

  • Ahsan

    assalam o alaikum

    can u guide me to use mysql connections instead of sqlite ?

    • Ahmed

      Hi Ahsan,
      You can make use of System.Data.MySqlClient namespace rather than System.Data.SQLite that i used in this tutorial. But you would have to download and install the ADO.net driver for MySQL first from: http://dev.mysql.com/downloads/connector/net/
      Once you have this setup, other parts of the data access tutorial should remain valid with the new namespace.

  • Rezell

    Hi Sir, can you please help me understand everything more. I’m having trouble on recognizing the face. Please, help…

  • Rezell

    Sir, where can i find the traineddata file? Or how can i make it? Please help..

    • Ahmed

      Hi,
      The trained data file is used to save features from the training set. You can take a look at the public bool TrainRecognizer() method and the _faceRecognizer.Save(_recognizerFilePath); method. The file is created where the recognizer file path points to.

      • Rezell

        I’m getting this error. Can you please help me fix it? Thank you so much Ahmed

        • Ahmed

          Are you running your VS in administrator mode?

          • Rezell

            Yes sir, it is now working.. But I have another problem, the predict function never returned -1.. It always say that it is recognized someone even though it is not registered in the database.. Please help, thank you..

          • I’m also having a similar problem. Even when the picture is of a blank wall, it appears to be recognizing a person from the training set. Possibly an issue with the logic?

          • I am also facing the same problem, I hope someone gets a solution, No matter what I do it always gives the same person

  • minhdoong37

    This project is very interesting. But, I have a problem:
    – if I use webcam (of my laptop) this project run very well
    – if I use other cameras, the ImageBox run verey very slowly (seem 1 frame / 2-3 seconds)

  • Average Joe

    Where did you declare

    picCapturedUser
    and frmSaveDialog;

  • Abdullah Dilawer

    Can You send the video Demo of your app ?

  • Muhammad Taha

    Bhai Great (y)

  • Wedad Anbtawi

    Hello, do you have human body detection tutorial?

    • MOeez Raja

      yes it is possible you can do it with the more better result and with less effort using a special device Kinect sensor. If need any type of help you can email me moeezhraja@hotmail.com

      • Arpan

        Bro I need to capture face and store it in MySql at the time of registration and after the successful login I want to verify the face using face recognition continuously. That means if any time it detects that the face in front of it is not matching with the face stored in the database it shows me an ERROR. Can you please help me to develop this? Waiting for your reply. Thank you

  • KHALED SALEH

    dear brother,

    i implemented the face detection (emgu 2.4.0) in vb.net, can i do recognition also in vb.net? how?

    your valued help will be really appreciated

  • Oussama Bensalem

    Salem
    I hope my message find you in peace. First of all I want to thank you for your Article it really helped me in my work. But I still have own problem:
    I tired to use EigenFaceRecognizer but it did not work so I switched to LPBHFaceRecognizer and what happened is that I’m always geting 0 return I tied every thnig but it did not work need your help

  • noman abbasi

    ..Can anyone send me complete code of face detection and recognition ..it’s my final year project and i my in big trouble ..plz help me … c# using emguCV

  • Arpan

    Hay @disqus_b76ujyHRLm:disqus thanks for the great post. Actually I need to capture face and store it in MySql at the time of registration and after the successful login I want to verify the face using face recognition continuously. That means if any time it detects that the face in front of it is not matching with the face stored in the database it shows me an ERROR. Can you please help me to develop this? Waiting for your reply. Thank you

  • Bjhulk Max

    Can u send me the code of face detection and recognition complete project bjhulkmax@gmail.com, coz difficult to understand it..help me plz

    • Code Programmer

      bro did u got that project