Homemade surveillance using Kinect

In previous articles I presented a home video surveillance system, the ThiefWatcher application. It is an extensible application that works combining several protocols, such as cameras, triggers, communication channels and storage systems. In this article I will show how to implement some of these protocols using the Microsoft Kinect sensor.

This is the link to the first article of the ThiewWatcher surveillance system series, where you can download this application and its source code. Part of that code is necessary to compile the project that accompanies this article, specifically, the WatcherCommons project.

In this another link you can download the source code of the KinectProtocol solution, or, if you prefer, you can download the KinectProtocol executable files. In the first case you can add the project to the ThiefWatcher solution, in the second case, simply unzip the files into the directory where you have placed the application executables. This application is written in C# using Visual Studio 2015.

You also have to install the Kinect SDK. The version will depend on the model of sensor that you have, in my implementation I have used version 2.0, which corresponds to the Kinect sensor for Xbox One. In case you have the model for Xbox 360, you will have to use an earlier version, and you will have to modify the Sensor class of the project, to adapt it to that version.

The Kinect sensor is very suitable for this application, since it not only provides a color and infrared camera, but also two different ways to detect intruders, the depth sensor and the detection of people's bodies. Therefore, in the project two different protocols are implemented, one for the camera, KinectCamera, and another one for the alarm trigger, KinectTrigger.

The installation of the protocols in the ThiefWatcher application is very simple. Once you have copied the libraries to the directory with the application executables, simply use the Install Protocol option from the File menu and select the KinectProtocol.dll class library. To create a new camera using the Kinect sensor, once the protocol is installed, use the New Camera option in the File menu and select Kinect Camera. This camera does not need any user, password or url, so you can leave these values blank. The only configuration it supports is the selection of infrared or color mode. You can have a color and an infrared camera working at once, since both cameras work at the same time in the sensor. If you want more complete instructions I recommend you visit the articles linked above about the application.

As for the alarm trigger protocol, KinectTrigger, configure it in the control panel of the application. As alarm protocol you have to select Kinect Trigger and provide the appropriate connection string.

There are two operation modes to trigger the alarm, one of them is body detection, which will only detect people, but it is very convenient if you have pets at home that can trigger proximity sensors, for example. In this case, you must indicate in the connection string source=body, no further configuration parameters are necessary.

The other mode uses the depth sensor of the camera, indicating source=depth in the connection string. This sensor assigns to each pixel of the image the distance in millimeters between that point and the sensor. In this case, what the program does is comparing each frame with the previous one, by means of a simple subtraction, and counting the points whose distance has changed more than a certain threshold. If this account is greater than a given percentage of the points of the image, the alarm is fired. This threshold is configured in the connection string with the thres parameter, for example, to put a threshold of 100, you have to use thres=100. The percentage of points is indicated by the sens parameter. For example, source=depth;thres=100;sens=15 sets a threshold of 100 and a percentage of 15% of the points.

The Sensor class

This class encapsulates all the functionality to deal with the Kinect sensor, so it is the only one that has to be adapted to use a different version of the SDK. It is a static class, so you do not have to instantiate it, and it only uses a single sensor, although in theory you could have several sensors connected at once.

The SensorCaps enum is used to indicate the type of inputs that must be processed, in order to optimize the work:

[Flags]
public enum SensorCaps
{
None = 0,
Color = 1,
Infrared = 2,
Depth = 4,
Body = 8
}

In the Sensor.Caps property you can indicate the appropriate combination of these values.

You have two properties for color images, Sensor.ColorFrameSize, which is read-only, which returns the size of the image as a Size struct, and the Sensor.ColorFrame property, which returns a Bitmap object with the last image captured.

infrared images are handled with two homologous properties: Sensor.InfraredFrameSize and Sensor.InfraredFrame.

The body detection is performed through the Sensor.BodyTracked Boolean property, and the depth sensor data is obtained, in the form of an ushort array, from the Sensor.DepthFrame property.

The sensor is put into operation using the OpenSensor method, which receives a parameter of SensorCaps type. As this method is called by each camera and by the alarm protocol and it is only necessary to start the sensor once, we keep an account of instances that have called the method in the global variable _instances:

public static void OpenSensor(SensorCaps caps)
{
Caps |= caps;
if (_sensor == null)
{
_instances = 1;
_sensor = KinectSensor.GetDefault();
if (!_sensor.IsOpen)
{
_sensor.Open();
}
Initialize();
_reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color
| FrameSourceTypes.Depth
| FrameSourceTypes.Infrared
| FrameSourceTypes.Body);
_reader.MultiSourceFrameArrived +=
new EventHandler<MultiSourceFrameArrivedEventArgs>(OnNewFrame);
}
else
{
_instances++;
}
}

The global variable _sensor, of KinectSensor type, will be null the first time this method is called. In this case we also create a MultiSourceFrameReader frame reader, in the _reader variable, which allows us to configure several types of data to be read at the same time. In this case we will always read color and infrared images, depth data and the list of detected bodies.

The way to read them will be through the MultiSourceFrameArrived event, which will trigger every time a new frame is available.

The method to stop an instance opened with OpenSensor is CloseSensor, which will reduce the count of instances until it reaches the last one, leaving in this case the system in its initial state and releasing resources:

public static void CloseSensor()
{
_instances--;
if (_instances <= 0)
{
if (_sensor != null)
{
if (_sensor.IsOpen)
{
_sensor.Close();
_reader.Dispose();
_reader = null;
_sensor = null;
}
}
_instances = 0;
Caps = SensorCaps.None;
}
}

KinectCamera and KinectTrigger classes

These classes implement the interfaces of the ThiefWatcher application that convert them into a camera protocol, IWatcherCamera and into an alarm protocol, ITrigger. For an explanation of these interfaces, I refer you to the article about the details of the configuration of each of the protocols and the cameras, of the original series of articles.

The task responsible for reading the cameras is implemented as follows:

private void CaptureThread(object interval)
{
while (_bCapturing)
{
try
{
DateTime tm = DateTime.Now;
Bitmap bmp = _nResolution == 0 ?
Sensor.ColorFrame :
Sensor.InfraredFrame;
if (bmp != null)
{
OnNewFrame?.Invoke(this, new FrameEventArgs(bmp));
TimeSpan ts;
do
{
ts = DateTime.Now.Subtract(tm);
} while (((ts.Seconds * 1000) +
ts.Milliseconds) <= (int)interval);
}
}
catch
{
}
}
}

You can see that the use of the Sensor class is trivial. The implementation of the alarm control task is also very simple:

private void TriggerTest()
{
while (_bRunning && (OnTriggerFired != null))
{
try
{
if (_bSource)
{
ushort[] frame = Sensor.DepthFrame;
if (frame != null)
{
if (_oldFrame != null)
{
int max = (_sens * frame.Length) / 100;
if (max > 0)
{
int cnt = 0;
for (int ix = 0; ix < frame.Length; ix++)
{
if (Math.Abs(frame[ix] - _oldFrame[ix]) >
_threshold)
{
cnt++;
if (cnt >= max)
{
OnTriggerFired?.Invoke(this,
EventArgs.Empty);
Thread.Sleep(2000);
break;
}
}
}
}
}
_oldFrame = frame;
}
}
else
{
if (Sensor.BodyTracked)
{
OnTriggerFired?.Invoke(this, EventArgs.Empty);
Thread.Sleep(2000);
}
}
}
catch
{
}
}
}

Where _bSource indicates the operating mode.