Posture recognition with Kinect IV

We continue with the series about posture recognition with the Microsoft Kinect sensor. This time I will comment the classes responsible of posture normalization of the body in the way commented in the first article, so that we obtain a series of data that constitute a simplified version of the skeleton most appropriate for analysis using pattern recognition techniques.

In this link you can find the first article in this series. In this another one you can download the source code of the KinectGestures solution, written in csharp with Visual Studio 2015.

The classes that carry on this normalization are in the KinectInterface assembly, in the KinectInterface.Normalization namespace. There are classes to normalize the forearms, arms, shins, legs and torso. In addition there is a class that performs a normalization of the entire skeleton, making use of all these classes for separate body parts.

The result, as I explained in the first article, consists of one or more values that represent the angles between the different bones. In the case of the forearm, there is only one angle, the one that forms the forearm with the arm. In the case of the arm, which also comprises the shoulder, three angles are provided, since the arm can freely rotate in a circle around the shoulder and also be more or less inclined with respect to the shoulder.

In total, for a complete skeleton, 20 different values are obtained for each position. Four values corresponding to the torso, three to each arm and leg, and one to each forearm and shin. The position of the head, feet and hands is not taken into account for normalization.

The NormalizerBase class

The NormalizerBase class is the abstract base class for all others. It consists of a variable _bones, an array of structures of type BodyPoint, and an optional identifier _id, of type string. The properties that it exposes are the following:

public BodyPoint[] NormalizedBones { get; }
public abstract int ZeroIndex { get; }
public string NormalizedPosition { get; protected set; }
protected abstract Vector3D Zero { get; }
protected abstract BodyPart Part { get; }

With NormalizedBones you get the normalized version of the joints that the class uses to calculate the angles, that is, with the bones aligned with the coordinate planes. ZeroIndex represents the index of the joint that is used as the center of rotation, that is, as the origin of coordinates. NormalizedPosition is the string with the normalized position, a series of integer values separated by semicolons. Zero and Part are used internally to indicate the coordinates of the joint that is at the origin and determine which part of the body is normalized by the class, respectively.

All the normalization process is done at the moment of creation of the instance of the class, in the constructor, by means of the Normalize method, which must be implemented by the derived classes:

public NormalizerBase(BodyVector bones, string id)
{
BodyPoint[] part = bones.GetBodyPart(Part);
_bones = new BodyPoint[part.Length];
_id = id;
part.CopyTo(_bones, 0);
Normalize();
}

The NormalizedAngle method is responsible for returning the value corresponding to a certain angle represented by the value of the sine and the cosine. The complete circle is divided into 20 sectors, so the returned values vary between 1 and 20. The other methods are used to perform rotations, inversions and displacements of the joints.

The ForearmNormalizer class

As an example of the implementation of a normalization class, we will see the ForearmNormalizer class, which has two subclasses derived, one for the right forearm and another for the left one. This class uses the wrist, elbow and shoulder joints to determine the angle that the forearm forms with the arm. The point that is used as the center of coordinates is the elbow.

The method where all the work is done is Normalize, which in this case is implemented as follows:

protected override void Normalize()
{
base.Normalize();
Vector3D prj = _bones[0].Vector.XYNormalizedProjection;
RotateZCC(prj.Y, prj.X);
prj = _bones[0].Vector.ZYNormalizedProjection;
RotateXCC(prj.Z, prj.Y);
prj = _bones[2].Vector.XYNormalizedProjection;
RotateZCC(prj.Y, prj.X);
prj = _bones[2].Vector.ZYNormalizedProjection;
NormalizedPosition = _id + NormalizedAngle(prj.Y, prj.Z);
}

We must call the base class method first to move the joints to the origin of coordinates. From here, it is about rotating the bones so that we leave them always aligned on the same plane to calculate the angle they form.

Using the methods of the Vector class, we obtain the sine and the cosine that forms the bone that we are going to use to rotate the whole set, by means of the UVNormalizedPosition method, where U and V are the coordinate axes that form the plane. Next, we will use the method RotateA.., where A represents the rotation axis and .. can be CC for a counter-clock turn or TC for a towards-clock one, to rotate around the other axis the angle determined by the sine and the cosine.

Finally, we assign to the property NormalizedPosition the angle or angles resulting from the operation.

The NormalizedBody class

The NormalizedBody class is an auxiliary class that allows us to perform the normalization of all body parts at once. In addition to providing properties with the standardized version of each of the parts separately, it offers the entire body data in the CompleteBody property.

This is the complete body normalization procedure, executed directly in the class constructor:

public NormalizedBody(BodyVector bv)
{
ForearmNormalizer normbf = new LeftForearmNormalizer(bv, "");
string s = normbf.NormalizedPosition;
LeftForearm = s;
normbf = new RightForearmNormalizer(bv, "");
s += ";" + normbf.NormalizedPosition;
RightForearm = normbf.NormalizedPosition;
ArmNormalizer normba = new LeftArmNormalizer(bv, "");
s += ";" + normba.NormalizedPosition;
LeftArm = normba.NormalizedPosition;
normba = new RightArmNormalizer(bv, "");
s += ";" + normba.NormalizedPosition;
RightArm = normba.NormalizedPosition;
ShinNormalizer normbs = new LeftShinNormalizer(bv, "");
s += ";" + normbs.NormalizedPosition;
LeftShin = normbs.NormalizedPosition;
normbs = new RightShinNormalizer(bv, "");
s += ";" + normbs.NormalizedPosition;
RightShin = normbs.NormalizedPosition;
LegNormalizer normbl = new LeftLegNormalizer(bv, "");
s += ";" + normbl.NormalizedPosition;
LeftLeg = normbl.NormalizedPosition;
normbl = new RightLegNormalizer(bv, "");
s += ";" + normbl.NormalizedPosition;
RightLeg = normbl.NormalizedPosition;
TorsoNormalizer normbt = new TorsoNormalizer(bv, "");
s += ";" + normbt.NormalizedPosition;
Torso = normbt.NormalizedPosition;
CompleteBody = s;
}

And this is all regarding normalization. In the next article I will finish this series showing the implementation of the main program, which uses all these classes to identify the position of the user's forearms.