Posture recognition with Kinect V

This article closes the series on posture recognition using the Microsoft Kinect sensor. To finish, I will simply show you the implementation of a basic form that uses the class libraries explained in the previous articles.

In this link you can find the first article in this series. In this another one you can download the source code of the KinectGestures solution, written in csharp with Visual Studio 2015.

The KinectGestures application

The KinectGestures project contains the Gestures form, with all the code needed to run the sample application:

It is a very simple form, consisting of a toolbar with a button to start the sensor and another one to stop it, a central panel on which the body is drawn and four panels surrounding it in order to indicate if the user is outing frame at any of the sides.

The form also uses a BackgroundWorker component to read the sensor data asynchronously, with a background process.

There are two local variables to control the sensor:

private ISensor _sensor = null;
private BodyVector _body = null;

The _sensor variable allows access to the Kinect sensor and the _body one contains the last body read.

The sensor is obtained in the form constructor using reflection. This is done in order to avoid having to refer to a specific version of the sensor library class, since it may be necessary to make several versions of that library to adapt it to the hardware version:

Assembly asm = Assembly.LoadFile(Path.Combine(
Path.GetDirectoryName(Application.ExecutablePath), "KTSensor.dll"));
foreach (Type t in asm.GetTypes())
{
if (t.GetInterface("ISensor") != null)
{
_sensor = t.GetConstructor(Type.EmptyTypes).Invoke(null) as ISensor;
break;
}
}

To start the sensor, you only have to call the Start method. The BackgroundWorker is also put into operation to collect the body frames:

try
{
_sensor.Start();
bPlay.Enabled = false;
bStop.Enabled = true;
_lastFrame = DateTime.Now;
bgBodyReader.RunWorkerAsync();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}

The BackgroundWorker executes a loop in which it simply gets the following body calling the NextBody method. Then it passes this body with the ReportProgress method to the ProgressChanged event, which runs in the same process as the form and will not cause problems when drawing the skeleton:

private void bgBodyReader_DoWork(object sender, DoWorkEventArgs e)
{
while (true)
{
if (bgBodyReader.CancellationPending)
{
e.Cancel = true;
break;
}
BodyVector bv = _sensor.NextBody;
if (bv != null)
{
bgBodyReader.ReportProgress(1, bv);
}
}
}

The ProgressChanged event is responsible for processing the skeleton:

private void bgBodyReader_ProgressChanged(object sender,
ProgressChangedEventArgs e)
{
if (e.UserState is BodyVector)
{
_body = e.UserState as BodyVector;
if (DateTime.Now.Subtract(_lastFrame).TotalMilliseconds >= 100)
{
_lastFrame = DateTime.Now;
NormalizedBody nb = new NormalizedBody(_body);
_normBody = nb.CompleteBody;
int lpos = int.Parse(nb.LeftForearm);
int rpos = int.Parse(nb.RightForearm);
if ((lpos >= 5) && (lpos <= 7) && (rpos >= 5) && (rpos <= 7))
{
pBody.BackColor = Color.Pink;
}
else if ((lpos >= 5) && (lpos <= 7))
{
pBody.BackColor = Color.Yellow;
}
else if ((rpos >= 5) && (rpos <= 7))
{
pBody.BackColor = Color.Cyan;
}
else
{
pBody.BackColor = Color.White;
}
Refresh();
}
}
}

The BodyVector object with the skeleton is found in the UserState property of the event's argument, of generic object type. We will only show one image every 100 milliseconds, to avoid excessive blinking. With the NormalizedBody class that we saw in previous articles we check the angle formed by the left and right forearms, so that we change the color of the background if we have one of them or both forming more or less a 90 degree angle with the arm.

To stop the capture, the execution of the BackgroundWorker is canceled. This cancellation triggers the RunWorkerCompleted event, where the sensor is stopped using the Stop method:

private void bgBodyReader_RunWorkerCompleted(object sender,
RunWorkerCompletedEventArgs e)
{
if (e.Error != null)
{
MessageBox.Show(e.Error.Message);
}
try
{
_sensor.Stop();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
bPlay.Enabled = true;
bStop.Enabled = false;
}
}

The drawing of the skeleton and the out of frame indications is done in the Paint event of the main panel of the form, using the auxiliary functions DrawBody and DrawBone, which each one can implement the way they want so that the skeleton looks as they like.