Mirroring 3D Character Animations

Introduction

Video games have resources. Resources are raw data that need to be manipulated, baked and become ready to be used in game. Textures, meshes, animations and sometimes metadatas are all counted as resources. These resources are consuming significant amount of memory. Re-using and manipulating resources is essential for a game engine.

In terms of animation, there exists plenty of actions which can be used to manage animations as resources and one is motion retargeting.

With motion retargeting, one can use a specific animation on different skeletons with different reference or binding poses, different joint size and different heights. For example, you just have a walk animation and want to use it for 5 different characters with different physical shapes. Motion retargeting systems can do this nicely so you don't need to have five different walks for those 5 different characters. You just have one walk animation which can be used for all characters. This represents lower amount of animations and therefore less needed resources.

Motion retargeting systems apply some modifications on top of animation data to make them suitable for different skeletons. These modifications include:

1- Defining a generic but modifiable skeleton template for bipeds or quadrupeds
2- Root motion reasonable scaling
3- Ability to edit skeleton reference pose
4- Joint movement limitations
5- Animation mirroring
6- Adding a run-time rig on top of the skeleton template.

Creating a motion retargeting system needs a vast amount of work and it's a huge topic. In this post I just want to show you how you can mirror character animations. Motion retargeting systems are usually supporting animation mirroring. It's useful for different purposes. Mirrored animations can be used to avoid foot-skating and also for achieving responsiveness and again, by mirroring an input pose, you can avoid creating new mirrored animations and you just using the same animation data, no new animation needed here. You can select the animation or its mirrored based on the foot phases.

In the next post, I will show you how you can use mirrored animations in action but this post is just concentrating on mirroring an input pose from an animation.

For this post, I used Unreal Engine 4. Unreal Engine has a very robust, flexible and optimized animation system but its motion retargeting is still immature. At this time, it can't be compared with Unity3D or Havok Animation motion retargeting.

Mirror Animations

To mirror animations, two types of bones should be considered. First the bones that have a mirrored bone in the skeleton hierarchy like hands, arms, legs, foots and facial bones. Let's call these mirrored bones, twins. Second, the bones which have no twin, like pelvis, spines, neck and head.

So to create a mirroring system, we have to define some meta data about the skeleton. It should save each bone twins, if it has any. For this reason, I define a class named AnimationMirrorData which saves and manipulate required data such as mirror-mapped bones, rotation mirror axis and position negation direction.

To mirror animations, I defined a custom animation node which can be used in unreal engine animation graph. It receives a pose in local space and mirrors it. It also has two input pins. One is for an animation mirror data object which should be initialized by the user and one is a boolean which let the node to be turned on or off. As you can see in the picture, there is no extra animation needed here and the node just accepts the current pose and mirrors it and you can turn it on or off based on the game or animation circumstances.

Here I discuss how to mirror each type of bones:

1- Mirroring bones which has a twin in the hierarchy

These kind of bones like hands and legs have a twin in the hierarchy. To mirror them, we need to swap the transforms of the two bones. For example the left upper arm transform should be pasted on the right upper arm, and the right upper arm transform should be pasted on the left upper arm. To do this, we have to deduct the the binding pose from the current transform of the bone at the current frame. In Unreal Engine 4 the local poses are calculated in their parent space as well as the binding poses. We don't want to mirror the binding poses of the bones and we just need to mirror the deducted transform. By doing this, we can make sure that the character can stay on the same spot and it won't rotate 180 degrees. Remember, this only works if the binding poses of the twin bones in the skeleton are already mirrored. This means that the rigger should have mirrored the twin bones when he/she wanted to rig the mesh.

2- Mirroring bones with no twin

These kind of bones are like root, pelvis or spine which don't have any twin in the hierarchy. For these kind of bones, again we have to deduct the binding pose from the current bone transform.  Now the current deducted transform should be mirrored. This time we need a mirror axis. The mirror axis should be selected by the user. Mostly it is x,y or z in the bone's binding pose space. So for rotations, if you select X as the mirror axis, you should negate the y and z components of the quaternion. To mirror the translations, things are a little different because for translations we never want to change the up and forward direction of the translations. That means by mirroring the animation, we don't want the character to move upside down and also backward. We just want the side movement to be negated. So here for the translations we just need to negate one component of the translation vector. So it is not counted as a mirror, mathematically.

Following, I placed some parts of the code which I wrote for the mirror animation node:

Here is  the AnimationMirrorData header file:

 #pragma once  
   
 #include "Object.h"  
 #include "AnimationMirrorData.generated.h"  
   
 /**  
  *   
  */  
 UENUM(BlueprintType)  
 enum class MirrorDir : uint8  
 {  
      None = 0,  
      X_Axis = 1,  
      Y_Axis = 2,  
      Z_Axis = 3  
 };  
   
   
 UCLASS(BlueprintType)  
 class ANIMATIONMIRRORING_API UAnimationMirrorData : public UObject  
 {  
 GENERATED_BODY()  
 public:  
   
      UAnimationMirrorData();  
   
      //Shows mirror axis. 0 = None, 1 = X, 2 = Y, 3 = Z   
      UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Mirror Animation")  
      MirrorDir MirrorAxis_Rot;  
   
      UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Mirror Animation")  
      MirrorDir RightAxis;  
   
      UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Mirror Animation")  
      MirrorDir PelvisMirrorAxis_Rot;  
   
      UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Mirror Animation")  
      MirrorDir PelvisRightAxis;  
   
      //Functions  
      UFUNCTION(BlueprintCallable, Category = "Mirror Animation")  
      void SetMirrorMappedBone(const FName bone_name, const FName mirror_bone_name);  
   
      UFUNCTION(BlueprintCallable, Category = "Mirror Animation")  
      FName GetMirroMappedBone(const FName bone_name) const;  
   
      TArray<FName> GetBoneMirrorDataStructure() const;  
   
 protected:  
      TArray<FName> mMirrorData; 

} 

And here are two functions which are mainly responsible to mirror animations:

/***********************************************/  
 void FAnimMirror::Evaluate(FPoseContext& Output)  
 {  
      mBasePose.Evaluate(Output);  
   
   
      if (!mAnimMirrorData)  
      {  
           return;  
      }  
   
      if (Output.AnimInstance)  
      {  
           TArray<FCompactPoseBoneIndex> lAr;  
           int32 lCurrentMirroredBoneInd = 0;  
           int32 lMirBoneCount = mAnimMirrorData->GetBoneMirrorDataStructure().Num();  
   
           //Mirror Mapped Bones  
           for (uint8 i = 0; i < lMirBoneCount; i += 2)  
           {  
                FCompactPoseBoneIndex lInd1 = FCompactPoseBoneIndex(Output.AnimInstance->GetSkelMeshComponent()->GetBoneIndex(mAnimMirrorData->GetBoneMirrorDataStructure()[i]));  
                FCompactPoseBoneIndex lInd2 = FCompactPoseBoneIndex(Output.AnimInstance->GetSkelMeshComponent()->GetBoneIndex(mAnimMirrorData->GetBoneMirrorDataStructure()[i + 1]));  
   
                FTransform lT1 = Output.Pose[lInd1];  
                FTransform lT2 = Output.Pose[lInd2];  
   
                Output.Pose[lInd1].SetRotation(Output.Pose.GetRefPose(lInd1).GetRotation() * Output.Pose.GetRefPose(lInd2).GetRotation().Inverse() * lT2.GetRotation());  
                Output.Pose[lInd2].SetRotation(Output.Pose.GetRefPose(lInd2).GetRotation() * Output.Pose.GetRefPose(lInd1).GetRotation().Inverse() * lT1.GetRotation());  
   
                Output.Pose[lInd1].SetLocation((Output.Pose.GetRefPose(lInd2).GetRotation().Inverse() * lT2.GetRotation() * (lT2.GetLocation() - Output.Pose.GetRefPose(lInd2).GetLocation()))   
                     + Output.Pose.GetRefPose(lInd1).GetLocation());  
                  
                Output.Pose[lInd2].SetLocation((Output.Pose.GetRefPose(lInd1).GetRotation().Inverse() * lT1.GetRotation() * (lT1.GetLocation() - Output.Pose.GetRefPose(lInd1).GetLocation()))   
                     + Output.Pose.GetRefPose(lInd2).GetLocation());  
   
                lAr.Add(lInd1);  
                lAr.Add(lInd2);  
   
           }  
   
   
           //Mirror Unmapped Bones  
           FCompactPoseBoneIndex lPoseBoneCount = FCompactPoseBoneIndex(Output.Pose.GetNumBones());  
   
           for (FCompactPoseBoneIndex i = FCompactPoseBoneIndex(0); i < lPoseBoneCount;)  
           {  
                if (!lAr.Contains(i))  
                {  
                     if (!i.IsRootBone())  
                     {  
                          FTransform lT = Output.Pose[i];  
                          lT.SetRotation(Output.Pose.GetRefPose(i).GetRotation().Inverse() * Output.Pose[i].GetRotation());  
                          lT.SetLocation(Output.Pose[i].GetLocation() - Output.Pose.GetRefPose(i).GetLocation());  
                            
                          if (i.GetInt() != 1)  
                          {  
                               MirrorPose(lT, (uint8)mAnimMirrorData->MirrorAxis_Rot, (uint8)mAnimMirrorData->RightAxis);  
                               Output.Pose[i].SetRotation(Output.Pose.GetRefPose(i).GetRotation() * lT.GetRotation());  
                               Output.Pose[i].SetLocation(Output.Pose.GetRefPose(i).GetLocation() + lT.GetLocation());  
                          }  
                          else  
                          {  
                               MirrorPose(lT, (uint8)mAnimMirrorData->PelvisMirrorAxis_Rot, (uint8)mAnimMirrorData ->PelvisRightAxis);  
                               Output.Pose[i].SetRotation(Output.Pose.GetRefPose(i).GetRotation() * lT.GetRotation());  
                               Output.Pose[i].SetLocation(Output.Pose.GetRefPose(i).GetLocation() + lT.GetLocation());  
                          }  
                     }  
                }  
                ++i;  
           }  
      }  
 };  
   
 void FAnimMirror::MirrorPose(FTransform& input_pose, const uint8 mirror_axis, const uint8 pos_fwd_mirror)  
 {  
   
      FVector lMirroredLoc = input_pose.GetLocation();  
   
      if (pos_fwd_mirror == 1)  
      {  
           lMirroredLoc.X = -lMirroredLoc.X;  
      }  
      else  
      {  
           if (pos_fwd_mirror == 2)  
           {  
                lMirroredLoc.Y = -lMirroredLoc.Y;  
           }  
           else  
           {  
                if (pos_fwd_mirror == 3)  
                {  
                     lMirroredLoc.Z = -lMirroredLoc.Z;  
                }  
           }  
      }  
   
      input_pose.SetLocation(lMirroredLoc);  
   
   
      switch (mirror_axis)  
      {  
           case 1:  
           {  
                const float lY = -input_pose.GetRotation().Y;  
                const float lZ = -input_pose.GetRotation().Z;  
                input_pose.SetRotation(FQuat(input_pose.GetRotation().X, lY, lZ, input_pose.GetRotation().W));  
                break;  
           }  
   
           case 2:  
           {  
                const  float lX = -input_pose.GetRotation().X;  
                const float lZ = -input_pose.GetRotation().Z;  
                input_pose.SetRotation(FQuat(lX, input_pose.GetRotation().Y, lZ, input_pose.GetRotation().W));  
                break;  
           }  
   
           case 3:  
           {  
                const float lX = -input_pose.GetRotation().X;  
                const float lY = -input_pose.GetRotation().Y;  
                input_pose.SetRotation(FQuat(lX, lY, input_pose.GetRotation().Z, input_pose.GetRotation().W));  
                break;  
           }  
      }  
 };  

I haven't placed the whole source code here. If you need them, just contact me and I will send them to you.

What Is a Binding Pose in Character Animation?

3D Character modelers always model 3D models in a pose known as T-pose where the character model stands in a certain pose and the hands and body shows a figure like T (Figure1)

     Figure1: A 3D model in T-pose and bones placed in it

Modelers model the characters in T-Pose so the riggers can rig the character easily. This helps them to place the bones to adjust vertex weights and envelopes easier. When all the bones are placed in the 3D model and the skinning process starts, the current pose of the skeleton is saved. This pose is called Binding Pose. The name speaks for itself. Binding pose is the pose where you start binding a mesh to its corresponding skeleton.

The idea of binding pose comes from Bio-mechanics where a human body motion is studied with respect to a pose called Standard Anatomical Position. Standard anatomical position is actually shows the rotation origin for each limb. Within this pose, the human body muscles has the highest level of rest hence it's the best pose to consider for the origin of the rotation of the limbs. That's the reason the 3D modelers model the characters in similar pose because the muscles are the least contracted and it's the most neutral pose for the body. Then the character riggers place the bones to the mesh to define the biding pose as the origin of the transformation of the bones.

Now why a binding pose should be saved?  You might know that all of the vertices' positions initially should be calculated in modeling space. A skinned mesh's vertices are transformed by its corresponding skeleton. When the skeleton bones transform, the vertices' positions change based on the weights they have for each bone (bones transformations are calculated in mesh's modeling space (object space)). Now if the vertices' positions change based on the orientation of the bones the whole mesh will be distorted because of an extra orientation or maybe position that the riggers applied to bones to adjust them into meshes. The rigger transforms each bone to fit them in the mesh's body. These transformations (call it Binding Pose Transformation) should not be calculated for the final vertices positions. You can consider an example in which you have a skinned mesh, so by changing bone transformations the vertices positions should be changed as well. Lets consider just one bone rotation. The orientation of the bone in mesh's modeling space is equal to:

Bone Orientation =  Binding Pose Rotation * KeyFrame Rotation

If vertices get affected by the "Bone Orientation" variable, first they will be rotated by Binding Pose Rotation and next with the key frame rotation and that binding pose rotation will distort all of the vertices because the initial model that the modeler created is affected by some extra unnecessary rotations produced by the rigger. So for bones belonging to a skeleton of a skinned mesh, a reference pose have to be saved and this reference pose is called Binding Pose. It will be saved when you apply a skin modifier to your mesh or in other words when you start the skinning process. For each bone in character animation, all of the key frames are stored relative to its binding pose. For a skinned mesh, deformations occur based on the relative transformations of each bone to its binding pose. This will prevent the mesh to be distorted.

For animation blending each bone has a weight. The weight is ranged between 0 and 1 which 1 means the full calculation of the key frame transformation and 0 means the binding pose. Now any other number between 0 and 1 for a weight shows a blended transformation between bones binding poses and key frames transformations.

As a side note, saving the keyframes relative to binding pose facilitates the process of  animation retargetting and also provides easier ways to compress the keyframes.