First Order Motion Model for Image Animation

Aliaksandr Siarohin
DISI, University of Trento
[email protected]

Stéphane Lathuilière
DISI, University of Trento
LTCI, Télécom Paris, Institut polytechnique de Paris
[email protected]

Sergey Tulyakov
Snap Inc.
[email protected]

Elisa Ricci
DISI, University of Trento
Fondazione Bruno Kessler
[email protected]

Nicu Sebe
DISI, University of Trento
Huawei Technologies Ireland
[email protected]

Abstract

Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available$^{1}$.

$^{1}$https://github.com/AliaksandrSiarohin/first-order-model

33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

Executive Summary: ### Context and Problem

Animating still images to create videos has wide applications in film production, photography, and online shopping, where users often want to see products or subjects move realistically without filming new content. The challenge lies in transferring motion from one video to an object in a static image, especially without detailed 3D models or manual labels for each object type, such as faces or human bodies. Traditional methods rely on custom models for specific objects, which are costly and limit flexibility. With recent advances in deep learning, there is growing demand for general approaches that work across object categories using only unlabelled videos, enabling faster and broader use in creative industries.

Objective

This paper introduces a new deep learning framework for image animation that generates video sequences by applying motion from a driving video to an object in a source image. The goal is to create an object-agnostic system—no pre-built models or annotations needed—that learns from videos of the same category and handles complex, real-world motions effectively.

Approach

The authors developed a self-supervised neural network trained on collections of videos showing objects like faces or human bodies, without any paired examples or labels. The system separates an object's appearance from its motion: it detects key feature points (called keypoints) and uses simple local adjustments (affine transformations) around them to capture movements, plus a mask to handle hidden parts (occlusions). During training, the model reconstructs videos by combining a single frame's appearance with motion from other frames, using datasets such as VoxCeleb (over 19,000 face videos), Tai-Chi-HD (a new high-resolution set of 2,800 human movement clips), UvA-Nemo (1,200 face videos), and BAIR (42,000 robot arm videos). At test time, it animates any source image from the trained category using motion from an unrelated driving video. Training took about two days on standard GPUs, with evaluations on video reconstruction quality and user preferences for animations.

Key Findings

The framework significantly outperformed prior methods like Monkey-Net and X2Face on video reconstruction tasks across all four datasets. For instance, on the challenging Tai-Chi-HD dataset with non-rigid human motions, it reduced average pixel error (L1) by about 18% compared to Monkey-Net (from 0.077 to 0.063) and keypoint distance error by roughly 36% (from 10.8 to 6.9 pixels). It also cut missing keypoint rates—indicating better visible part generation—by over 40% in complex poses. On face datasets like VoxCeleb, errors dropped 12-30%, and it handled higher resolutions (256x256 pixels) where competitors struggled. Ablation tests confirmed the value of local transformations and occlusion handling: without them, errors rose 10-20%, and animations blurred or failed on large poses. In user studies with 50 animation pairs, 70-90% preferred outputs from this model over baselines, especially for body movements.

Implications and Interpretation

These results show the method produces more realistic and detailed animations by better capturing subtle deformations and hidden areas, unlike simpler prior approaches that treat motion as rigid shifts and distort objects in big changes. This reduces artifacts in videos, improving usability for applications like virtual try-ons in e-commerce or quick edits in movies, potentially cutting production time and costs by avoiding custom training per object. It also advances self-supervised learning, proving compact motion representations (just 10 keypoints) can rival larger models without annotations. Unexpectedly, it excelled on diverse categories beyond faces, like robots, suggesting broader potential, though it assumes rough pose similarity between source and driving inputs for best results.

Recommendations and Next Steps

Adopt this framework for prototyping animation tools, starting with face or body categories, as the open-source code and new Tai-Chi-HD dataset make integration straightforward. For production, prioritize relative motion transfer to preserve original proportions, trading off some flexibility for naturalness. Explore expanding to new categories like animals or vehicles via retraining on existing video collections. Before full deployment, run pilots on specific use cases to fine-tune for speed. Further work should include more diverse datasets for edge cases and optimizations for real-time video processing on mobile devices.

Limitations and Confidence

The model relies on videos from the same category for training, so it may underperform on very dissimilar objects or extreme poses without initial alignment. Occlusion detection, while effective, can sometimes over-inpaint simple scenes. Overall confidence is high, backed by consistent quantitative gains and user preferences across benchmarks, but caution is advised for applications needing perfect anatomical accuracy, where additional validation data could help.

1. Introduction

Section Summary: Image animation involves creating videos by adding motion from a driving video to objects in a still source image, with applications in movies, photography, and online shopping, such as making one person's face mimic another's expressions. Traditional methods rely on detailed 3D models tailored to specific objects, while newer deep learning techniques like GANs and VAEs offer more flexibility but often need expensive pre-trained tools that don't work for all objects. To improve this, the authors introduce a versatile approach using self-learned keypoints and local transformations to handle complex motions and occlusions, which outperforms existing methods on high-resolution videos and includes a new benchmark dataset called Thai-Chi-HD.

Generating videos by animating objects in still images has countless applications across areas of interest including movie production, photography, and e-commerce. More precisely, image animation refers to the task of automatically synthesizing videos by combining the appearance extracted from a source image with motion patterns derived from a driving video. For instance, a face image of a certain person can be animated following the facial expressions of another individual (see Figure 1). In the literature, most methods tackle this problem by assuming strong priors on the object representation (e.g. 3D model) [1] and resorting to computer graphics techniques [2, 3]. These approaches can be referred to as object-specific methods, as they assume knowledge about the model of the specific object to animate.

Recently, deep generative models have emerged as effective techniques for image animation and video retargeting [4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. In particular, Generative Adversarial Networks (GANs) [14] and Variational Auto-Encoders (VAEs) [15] have been used to transfer facial expressions [10] or motion patterns [6] between human subjects in videos. Nevertheless, these approaches usually rely on pre-trained models in order to extract object-specific representations such as keypoint locations. Unfortunately, these pre-trained models are built using costly ground-truth data annotations [4, 8, 12] and are not available in general for an arbitrary object category. To address this issues, recently Siarohin et al. [9] introduced Monkey-Net, the first object-agnostic deep model for image animation. Monkey-Net encodes motion information via keypoints learned in a self-supervised fashion. At test time, the source image is animated according to the corresponding keypoint trajectories estimated in the driving video. The major weakness of Monkey-Net is that it poorly models object appearance transformations in the keypoint neighborhoods assuming a zeroth order model (as we show in Section 3.1). This leads to poor generation quality in the case of large object pose changes (see Figure 4). To tackle this issue, we propose to use a set of self-learned keypoints together with local affine transformations to model complex motions. We therefore call our method a first-order motion model. Second, we introduce an occlusion-aware generator, which adopts an occlusion mask automatically estimated to indicate object parts that are not visible in the source image and that should be inferred from the context. This is especially needed when the driving video contains large motion patterns and occlusions are typical. Third, we extend the equivariance loss commonly used for keypoints detector training [16, 17], to improve the estimation of local affine transformations. Fourth, we experimentally show that our method significantly outperforms state-of-the-art image animation methods and can handle high-resolution datasets where other approaches generally fail. Finally, we release a new high resolution dataset, Thai-Chi-HD, which we believe could become a reference benchmark for evaluating frameworks for image animation and video generation.

$^{2}$https://aliaksandrsiarohin.github.io/first-order-model-website/

2. Related work

Section Summary: Earlier research on video generation explored using neural networks to create video frames from random noise, evolving into conditional methods that produce videos based on labels, images, or initial frames, often by warping existing footage. In image animation, traditional techniques relied on specific object models like 3D faces or required extensive labeled training data per individual for translating motion between videos, but newer approaches avoid such priors by using motion fields or sparse keypoints to warp and generate animated images. The presented method builds on these by employing self-supervised keypoints with local transformations and explicit handling of occlusions, enabling simpler, more flexible animation of various objects without needing reference poses or per-object training.

Video Generation. Earlier works on deep video generation discussed how spatio-temporal neural networks could render video frames from noise vectors [19, 20]. More recently, several approaches tackled the problem of conditional video generation. For instance, Wang et al. [21] combine a recurrent neural network with a VAE in order to generate face videos. Considering a wider range of applications, Tulyakov et al. [22] introduced MoCoGAN, a recurrent architecture adversarially trained in order to synthesize videos from noise, categorical labels or static images. Another typical case of conditional generation is the problem of future frame prediction, in which the generated video is conditioned on the initial frame [23, 24, 25, 26, 17]. Note that in this task, realistic predictions can be obtained by simply warping the initial video frame [27, 23, 26]. Our approach is closely related to these previous works since we use a warping formulation to generate video sequences. However, in the case of image animation, the applied spatial deformations are not predicted but given by the driving video.

Image Animation. Traditional approaches for image animation and video re-targeting [2, 3, 28] were designed for specific domains such as faces [29, 7], human silhouettes [30, 10, 8] or gestures [12] and required a strong prior of the animated object. For example, in face animation, method of Zollhofer et al. [29] produced realistic results at expense of relying on a 3D morphable model of the face. In many applications, however, such models are not available. Image animation can also be treated as a translation problem from one visual domain to another. For instance, Wang et al. [10] transferred human motion using the image-to-image translation framework of Isola et al. [31]. Similarly, Bansal et al. [6] extended conditional GANs by incorporating spatio-temporal cues in order to improve video translation between two given domains. Such approaches in order to animate a single person require hours of videos of that person labelled with semantic information, and therefore have to be retrained for each individual. In contrast to these works, we neither rely on labels, prior information about the animated objects, nor on specific training procedures for each object instance. Furthermore, our approach can be applied to any object within the same category (e.g., faces, human bodies, robot arms etc).

Several approaches were proposed that do not require priors about the object. X2Face [11] uses a dense motion field in order to generate the output video via image warping. Similarly to us they employ a reference pose that is used to obtain a canonical representation of the object. In our formulation, we do not require an explicit reference pose, leading to significantly simpler optimization and improved image quality. Siarohin et al. [9] introduced Monkey-Net, a self-supervised framework for animating arbitrary objects by using sparse keypoint trajectories. In this work, we also employ sparse trajectories induced by self-supervised keypoints. However, we model object motion in the neighbourhood of each predicted keypoint by a local affine transformation. Additionally, we explicitly model occlusions in order to indicate to the generator network the image regions that can be generated by warping the source image and the occluded areas that need to be inpainted.

3. Method

Section Summary: This method animates an object from a single source image by applying the motion captured in a separate driving video, using a self-supervised approach trained on collections of videos featuring similar objects. The model learns to represent motion through sparse keypoints and local affine transformations derived from video frame pairs, then generates a dense motion field and occlusion map to map movements from the driving video to the source image. At test time, it warps the source image according to this motion and fills in any hidden parts to create the animated output.

We are interested in animating an object depicted in a source image $\mathbf{S}$ based on the motion of a similar object in a driving video $\mathcal{D}$. Since direct supervision is not available (pairs of videos in which objects move similarly), we follow a self-supervised strategy inspired from Monkey-Net [9]. For training, we employ a large collection of video sequences containing objects of the same object category. Our model is trained to reconstruct the training videos by combining a single frame and a learned latent representation of the motion in the video. Observing frame pairs, each extracted from the same video, it learns to encode motion as a combination of motion-specific keypoint displacements and local affine transformations. At test time we apply our model to pairs composed of the source image and of each frame of the driving video and perform image animation of the source object.

An overview of our approach is presented in Figure 2. Our framework is composed of two main modules: the motion estimation module and the image generation module. The purpose of the motion estimation module is to predict a dense motion field from a frame $\mathbf{D}\in\mathbb{R}^{3\times H\times W}$ of dimension $H\times W$ of the driving video $\mathcal{D}$ to the source frame $\mathbf{S}\in\mathbb{R}^{3\times H\times W}$. The dense motion field is later used to align the feature maps computed from $\mathbf{S}$ with the object pose in $\mathbf{D}$. The motion field is modeled by a function $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}} : \mathbb{R}^2 \rightarrow \mathbb{R}^2$ that maps each pixel location in $\mathbf{D}$ with its corresponding location in $\mathbf{S}$. $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}$ is often referred to as backward optical flow. We employ backward optical flow, rather than forward optical flow, since back-warping can be implemented efficiently in a differentiable manner using bilinear sampling [32]. We assume there exists an abstract reference frame $\mathbf{R}$. We independently estimate two transformations: from $\mathbf{R}$ to $\mathbf{S}$ ($\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}$) and from $\mathbf{R}$ to $\mathbf{D}$ ($\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}$). Note that unlike X2Face [11] the reference frame is an abstract concept that cancels out in our derivations later. Therefore it is never explicitly computed and cannot be visualized. This choice allows us to independently process $\mathbf{D}$ and $\mathbf{S}$. This is desired since, at test time the model receives pairs of the source image and driving frames sampled from a different video, which can be very different visually. Instead of directly predicting $\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}$ and $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}$, the motion estimator module proceeds in two steps.

In the first step, we approximate both transformations from sets of sparse trajectories, obtained by using keypoints learned in a self-supervised way. The locations of the keypoints in $\mathbf{D}$ and $\mathbf{S}$ are separately predicted by an encoder-decoder network. The keypoint representation acts as a bottleneck resulting in a compact motion representation. As shown by Siarohin et al. [9], such sparse motion representation is well-suited for animation as at test time, the keypoints of the source image can be moved using the keypoints trajectories in the driving video. We model motion in the neighbourhood of each keypoint using local affine transformations. Compared to using keypoint displacements only, the local affine transformations allow us to model a larger family of transformations. We use Taylor expansion to represent $\mathcal{T}_{\mathbf{D} \leftarrow \mathbf{R}}$ by a set of keypoint locations and affine transformations. To this end, the keypoint detector network outputs keypoint locations as well as the parameters of each affine transformation.

During the second step, a dense motion network combines the local approximations to obtain the resulting dense motion field $\hat{\mathcal{T}}{\mathbf{S} \leftarrow \mathbf{D}}$. Furthermore, in addition to the dense motion field, this network outputs an occlusion mask $\hat{\mathcal{O}}{\mathbf{S} \leftarrow \mathbf{D}}$ that indicates which image parts of $\mathbf{D}$ can be reconstructed by warping of the source image and which parts should be inpainted, i.e. inferred from the context.

Finally, the generation module renders an image of the source object moving as provided in the driving video. Here, we use a generator network $G$ that warps the source image according to $\hat{\mathcal{T}}_{\mathbf{S} \leftarrow \mathbf{D}}$ and inpaints the image parts that are occluded in the source image. In the following sections we detail each of these step and the training procedure.

3.1 Local Affine Transformations for Approximate Motion Description

The motion estimation module estimates the backward optical flow $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}$ from a driving frame $\mathbf{D}$ to the source frame $\mathbf{S}$. As discussed above, we propose to approximate $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}$ by its first order Taylor expansion in a neighborhood of the keypoint locations. In the rest of this section, we describe the motivation behind this choice, and detail the proposed approximation of $\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{D}}$.

We assume there exist an abstract reference frame $\mathbf{R}$. Therefore, estimating $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}$ consists in estimating $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}$ and $\mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}$. Furthermore, given a frame $\mathbf{X}$, we estimate each transformation $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}$ in the neighbourhood of the learned keypoints. Formally, given a transformation $\mathcal{T}_{\mathbf{X} \leftarrow \mathbf{R}}$, we consider its first order Taylor expansions in $K$ keypoints $p_1, \dots p_K$. Here, $p_1, \dots p_K$ denote the coordinates of the keypoints in the reference frame $\mathbf{R}$. Note that for the sake of simplicity in the following the point locations in the reference pose space are all denoted by $p$ while the point locations in the $\mathbf{X}$, $\mathbf{S}$ or $\mathbf{D}$ pose spaces are denoted by $z$. We obtain:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) = \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p_k) + \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)(p - p_k) + o(\Vert p - p_k \Vert),\tag{1} $

In this formulation, the motion function $\mathcal{T}_{\mathbf{X} \leftarrow \mathbf{R}}$ is represented by its values in each keypoint $p_k$ and its Jacobians computed in each $p_k$ location:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \simeq \left{ \left{\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p_1), \frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_1}\right}, \dots \left{\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p_k), \frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert_{p=p_K}\right}\right}.\tag{2} $

Furthermore, in order to estimate $\mathcal{T}{\mathbf{R} \leftarrow \mathbf{X}}=\mathcal{T}^{-1}{\mathbf{X} \leftarrow \mathbf{R}}$, we assume that $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}$ is locally bijective in the neighbourhood of each keypoint. We need to estimate $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}$ near the keypoint $z_k$ in $\mathbf{D}$, given that $z_k$ is the pixel location corresponding to the keypoint location $p_k$ in $\mathbf{R}$. To do so, we first estimate the transformation $\mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}$ near the point $z_k$ in the driving frame $\mathbf{D}$, e.g. $p_k = \mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}(z_k)$. Then we estimate the transformation $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}$ near $p_k$ in the reference $\mathbf{R}$. Finally $ \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}$ is obtained as follows:

$ \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}} = \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}} \circ \mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}} = \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}} \circ \mathcal{T}^{-1}_{\mathbf{D} \leftarrow \mathbf{R}},\tag{3} $

After computing again the first order Taylor expansion of Equation 3 (see Sup. Mat.), we obtain:

$ \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z) \approx \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k) + J_k (z - \mathcal{T}_{\mathbf{D} \leftarrow \mathbf{R}}(p_k))\tag{4} $

with:

$ J_k=\left(\frac{d}{dp}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)^{-1}\tag{5} $

In practice, $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k)$ and $\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p_k)$ in Equation 4 are predicted by the keypoint predictor. More precisely, we employ the standard U-Net architecture that estimates $K$ heatmaps, one for each keypoint. The last layer of the decoder uses softmax activations in order to predict heatmaps that can be interpreted as keypoint detection confidence map. Each expected keypoint location is estimated using the average operation as in [9, 33]. Note if we set $J_k=\mathbb{1}$ ($\mathbb{1}$ is $2\times2$ identity matrix), we get the motion model of Monkey-Net. Therefore Monkey-Net uses a zeroth-order approximation of $\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{D}}(z) - z$.

For both frames $\mathbf{S}$ and $\mathbf{D}$, the keypoint predictor network also outputs four additional channels for each keypoint. From these channels, we obtain the coefficients of the matrices $\frac{d}{dp}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p)\vert{p=p_k}$ and $\frac{d}{dp}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p) \vert{p=p_k}$ in Equation 5 by computing spatial weighted average using as weights the corresponding keypoint confidence map.

Combining Local Motions. We employ a convolutional network $P$ to estimate $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$ from the set of Taylor approximations of $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z)$ in the keypoints and the original source frame $\mathbf{S}$. Importantly, since $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$ maps each pixel location in $\mathbf{D}$ with its corresponding location in $\mathbf{S}$, the local patterns in $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$, such as edges or texture, are pixel-to-pixel aligned with $\mathbf{D}$ but not with $\mathbf{S}$. This misalignment issue makes the task harder for the network to predict $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$ from $\mathbf{S}$. In order to provide inputs already roughly aligned with $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$, we warp the source frame $\mathbf{S}$ according to local transformations estimated in Equation 4. Thus, we obtain $K$ transformed images $\mathbf{S}^1, \dots \mathbf{S}^K$ that are each aligned with $\mathcal{\hat{T}}_{\mathbf{S} \leftarrow \mathbf{D}}$ in the neighbourhood of a keypoint. Importantly, we also consider an additional image $\mathbf{S}^0 = \mathbf{S}$ for the background.

For each keypoint $p_k$ we additionally compute heatmaps $\mathbf{H}k$ indicating to the dense motion network where each transformation happens. Each $\mathbf{H}k(z)$ is implemented as the difference of two heatmaps centered in $\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p_k)$ and $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k)$:

$ \mathbf{H}k(z) = exp\left(\frac{\left(\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p_k) - z\right)^2}{\sigma}\right) - exp\left(\frac{\left(\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{R}}(p_k) - z\right)^2}{\sigma}\right). $

In all our experiments, we employ $\sigma=0.01$ following Jakab et al. [16].

The heatmaps $\mathbf{H}k$ and the transformed images $\mathbf{S}^0, \dots \mathbf{S}^K$ are concatenated and processed by a U-Net [34]. $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$ is estimated using a part-based model inspired by Monkey-Net [9]. We assume that an object is composed of $K$ rigid parts and that each part is moved according to 4. Therefore we estimate $K$ +1 masks $\mathbf{M}k, k=0, \dots K$ that indicate where each local transformation holds. The final dense motion prediction $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}(z)$ is given by:

$ \mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}(z) = \mathbf{M}0z + \sum{k=1}^K{ \mathbf{M}k \left(\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k) + J_k (z - \mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p_k))\right) } $

Note that, the term $\mathbf{M}_0z$ is considered in order to model non-moving parts such as background.

3.2 Occlusion-aware Image Generation

As mentioned in Section 3, the source image $\mathbf{S}$ is not pixel-to-pixel aligned with the image to be generated $\hat{\mathbf{D}}$. In order to handle this misalignment, we use a feature warping strategy similar to [35, 9, 36]. More precisely, after two down-sampling convolutional blocks, we obtain a feature map $\boldsymbol{\xi}\in\mathbb{R}^{H'\times W'}$ of dimension $H'\times W'$. We then warp $\boldsymbol{\xi}$ according to $\mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}}$. In the presence of occlusions in $\mathbf{S}$, optical flow may not be sufficient to generate $\hat{\mathbf{D}}$. Indeed, the occluded parts in $\mathbf{S}$ cannot be recovered by image-warping and thus should be inpainted. Consequently, we introduce an occlusion map $\mathcal{\hat{O}}{\mathbf{S} \leftarrow \mathbf{D}}\in[0, 1]^{H'\times W'}$ to mask out the feature map regions that should be inpainted. Thus, the occlusion mask diminishes the impact of the features corresponding to the occluded parts. The transformed feature map is written as:

$ \boldsymbol{\xi}'=\mathcal{\hat{O}}{\mathbf{S} \leftarrow \mathbf{D}}\odot f_w(\boldsymbol{\xi}, \mathcal{\hat{T}}{\mathbf{S} \leftarrow \mathbf{D}})\tag{6} $

where $f_w(\cdot, \cdot)$ denotes the back-warping operation and $\odot$ denotes the Hadamard product. We estimate the occlusion mask from our sparse keypoint representation, by adding a channel to the final layer of the dense motion network. Finally, the transformed feature map $\boldsymbol{\xi}'$ is fed to subsequent network layers of the generation module (see Sup. Mat.) to render the sought image.

3.3 Training Losses

We train our system in an end-to-end fashion combining several losses. First, we use the reconstruction loss based on the perceptual loss of Johnson et al. [37] using the pre-trained VGG-19 network as our main driving loss. The loss is based on implementation of Wang et al. [10]. With the input driving frame $\mathbf{D}$ and the corresponding reconstructed frame $\mathbf{\hat{D}}$, the reconstruction loss is written as:

$ L_{rec}(\mathbf{\hat{D}}, \mathbf{D}) = \sum_{i=1}^{I} \left|N_i(\mathbf{\hat{D}}) - N_i(\mathbf{D}) \right|, $

where $N_i(\cdot)$ is the $i^{th}$ channel feature extracted from a specific VGG-19 layer and $I$ is the number of feature channels in this layer. Additionally we propose to use this loss on a number of resolutions, forming a pyramid obtained by down-sampling $\mathbf{\hat{D}}$ and $\mathbf{D}$, similarly to MS-SSIM [38, 39]. The resolutions are $256\times256$, $128\times128$, $64\times64$ and $32\times32$. There are 20 loss terms in total.

Imposing Equivariance Constraint. Our keypoint predictor does not require any keypoint annotations during training. This may lead to unstable performance. Equivariance constraint is one of the most important factors driving the discovery of unsupervised keypoints [16, 40]. It forces the model to predict consistent keypoints with respect to known geometric transformations. We use thin plate splines deformations as they were previously used in unsupervised keypoint detection [16, 40] and are similar to natural image deformations. Since our motion estimator does not only predict the keypoints, but also the Jacobians, we extend the well-known equivariance loss to additionally include constraints on the Jacobians.

We assume that an image $\mathbf{X}$ undergoes a known spatial deformation $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}$. In this case $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}$ can be an affine transformation or a thin plane spline deformation. After this deformation we obtain a new image $\mathbf{Y}$. Now by applying our extended motion estimator to both images, we obtain a set of local approximations for $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}$ and $\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}$. The standard equivariance constraint writes as:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}} \equiv \mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}_{\mathbf{Y} \leftarrow \mathbf{R}} $

After computing the first order Taylor expansions of both sides, we obtain the following constraints (see derivation details in Sup. Mat.):

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p_k) \equiv \mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}_{\mathbf{Y} \leftarrow \mathbf{R}} (p_k),\tag{7} $

$ \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \equiv \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}(p) \middle\vert{p=\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p_k)} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p) \middle\vert_{p=p_k} \right),\tag{8} $

Note that the constraint Equation 7 is strictly the same as the standard equivariance constraint for the keypoints [16, 40]. During training, we constrain every keypoint location using a simple $L_1$ loss between the two sides of Equation 7. However, implementing the second constraint from Equation 8 with $L_1$ would force the magnitude of the Jacobians to zero and would lead to numerical problems. To this end, we reformulate this constraint in the following way:

$ \mathbb{1} \equiv \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)^{-1} \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}(p) \middle\vert{p=\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p_k)} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p) \middle\vert_{p=p_k} \right),\tag{9} $

where $\mathbb{1}$ is $2 \times 2$ identity matrix. Then, $L_1$ loss is employed similarly to the keypoint location constraint. Finally, in our preliminary experiments, we observed that our model shows low sensitivity to the relative weights of the reconstruction and the two equivariance losses. Therefore, we use equal loss weights in all our experiments.

3.4 Testing Stage: Relative Motion Transfer

At this stage our goal is to animate an object in a source frame $\mathbf{S}_1$ using the driving video $\mathbf{D}_1, \dots \mathbf{D}_T$. Each frame $\mathbf{D}_t$ is independently processed to obtain $\mathbf{S}t$. Rather than transferring the motion encoded in $\mathcal{T}{\mathbf{S}_1 \leftarrow \mathbf{D}_t}(p_k)$ to $\mathbf{S}$, we transfer the relative motion between $\mathbf{D}_1$ and $\mathbf{D}_t$ to $\mathbf{S}1$. In other words, we apply a transformation $\mathcal{T}{\mathbf{D}_t \leftarrow \mathbf{D}_1}(p)$ to the neighbourhood of each keypoint $p_k$:

$ \mathcal{T}_{\mathbf{S}1 \leftarrow \mathbf{S}t}(z) \approx \mathcal{T}{\mathbf{S}1 \leftarrow \mathbf{R}}(p_k) + J_k (z - \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k) + \mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p_k) - \mathcal{T}{\mathbf{D}_t \leftarrow \mathbf{R}}(p_k))\tag{10} $

with

$ J_k=\left(\frac{d}{dp}\mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{D}t \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)^{-1}\tag{11} $

Detailed mathematical derivations are provided in Sup. Mat.. Intuitively, we transform the neighbourhood of each keypoint $p_k$ in $\mathbf{S}_1$ according to its local deformation in the driving video. Indeed, transferring relative motion over absolute coordinates allows to transfer only relevant motion patterns, while preserving global object geometry. Conversely, when transferring absolute coordinates, as in X2Face [11], the generated frame inherits the object proportions of the driving video. It's important to note that one limitation of transferring relative motion is that we need to assume that the objects in $\mathbf{S}_1$ and $\mathbf{D}_1$ have similar poses (see [9]). Without initial rough alignment, Equation 10 may lead to absolute keypoint locations physically impossible for the object of interest.

4. Experiments

Section Summary: The experiments test the method on four datasets: VoxCeleb and UvA-Nemo for faces, BAIR for robot interactions with objects, and a new high-definition Tai-Chi-HD collection of human movement videos, all processed to 256x256 resolution for better quality than previous work. Evaluation uses video reconstruction as a stand-in task, measuring errors in pixel differences, keypoint motion accuracy, detection reliability, and image features, alongside user preference studies for animation realism. Results from ablation tests and user surveys show the full model outperforming simpler versions and rivals like X2Face and Monkey-Net, with up to 95% user favor on some datasets.

Datasets. We train and test our method on four different datasets containing various objects. Our model is capable of rendering videos of much higher resolution compared to [9] in all our experiments.

• The VoxCeleb dataset [18] is a face dataset of 22496 videos, extracted from YouTube videos. For pre-processing, we extract an initial bounding box in the first video frame. We track this face until it is too far away from the initial position. Then, we crop the video frames using the smallest crop containing all the bounding boxes. The process is repeated until the end of the sequence. We filter out sequences that have resolution lower than $256 \times 256$ and the remaining videos are resized to $256 \times 256$ preserving the aspect ratio. It's important to note that compared to X2Face [11], we obtain more natural videos where faces move freely within the bounding box. Overall, we obtain 19522 training videos and 525 test videos, with lengths varying from 64 to 1024 frames.

\begin{tabular}{c|ccc}
        \toprule
      & \multicolumn{3}{c}{\emph{Tai-Chi-HD}}  \\
      & $\mathcal{L}_1$  & ({AKD}, {MKR}) & {AED} \\ \toprule
        \emph{Baseline} & 0.073 & (8.945, 0.099) &  0.235 \\
        \emph{Pyr.} & 0.069 & (9.407, 0.065) & 0.213 \\
        \emph{Pyr.}+ $\mathcal{O}_{\mathbf{S} \leftarrow \mathbf{D}}$  & 0.069 & (8.773, 0.050) & 0.205 \\
        \emph{Jac. w/o Eq. (12)} & 0.073 & (9.887, 0.052) & 0.220 \\
        \emph{Full}& \bf 0.063 & (\bf 6.862, \bf 0.036)  & \bf 0.179 \\
        \bottomrule

    \end{tabular}

\begin{tabular}{c|cc}
\toprule
      & X2Face [11] & Monkey-Net [9] \\
\midrule
        \emph{Tai-Chi-HD}& 92.0\% & 80.6\%  \\
        \emph{VoxCeleb} & 95.8\% & 68.4\%\\
         \emph{Nemo}& 79.8\% & 60.6\%  \\
        \emph{Bair}& 95.0\% & 67.0\%\\
        \bottomrule
    \end{tabular}

• The UvA-Nemo dataset [41] is a facial analysis dataset that consists of 1240 videos. We apply the exact same pre-processing as for VoxCeleb. Each video starts with a neutral expression. Similar to Wang et al. [21], we use 1116 videos for training and 124 for evaluation.
• The BAIR robot pushing dataset [42] contains videos collected by a Sawyer robotic arm pushing diverse objects over a table. It consists of 42880 training and 128 test videos. Each video is 30 frame long and has a $256\times256$ resolution.
• Following Tulyakov et al. [22], we collected 280 tai-chi videos from YouTube. We use 252 videos for training and 28 for testing. Each video is split in short clips as described in pre-processing of VoxCeleb dataset. We retain only high quality videos and resized all the clips to $256\times256$ pixels (instead of $64\times64$ pixels in [22]). Finally, we obtain 3049 and 285 video chunks for training and testing respectively with video length varying from 128 to 1024 frames. This dataset is referred to as the Tai-Chi-HD dataset. The dataset will be made publicly available.

Evaluation Protocol. Evaluating the quality of image animation is not obvious, since ground truth animations are not available. We follow the evaluation protocol of Monkey-Net [9]. First, we quantitatively evaluate each method on the "proxy" task of video reconstruction. This task consists of reconstructing the input video from a representation in which appearance and motion are decoupled. In our case, we reconstruct the input video by combining the sparse motion representation in Equation 2 of each frame and the first video frame. Second, we evaluate our model on image animation according to a user-study. In all experiments we use $K$ =10 as in [9]. Other implementation details are given in Sup. Mat.

Metrics. To evaluate video reconstruction, we adopt the metrics proposed in Monkey-Net [9]:

• $\mathcal{L}_1$. We report the average $\mathcal{L}_1$ distance between the generated and the ground-truth videos.
• Average Keypoint Distance (AKD). For the Tai-Chi-HD, VoxCeleb and Nemo datasets, we use 3rd-party pre-trained keypoint detectors in order to evaluate whether the motion of the input video is preserved. For the VoxCeleb and Nemo datasets we use the facial landmark detector of Bulat et al. [43]. For the Tai-Chi-HD dataset, we employ the human-pose estimator of Cao et al. [44]. These keypoints are independently computed for each frame. AKD is obtained by computing the average distance between the detected keypoints of the ground truth and of the generated video.
• Missing Keypoint Rate (MKR). In the case of Tai-Chi-HD, the human-pose estimator returns an additional binary label for each keypoint indicating whether or not the keypoints were successfully detected. Therefore, we also report the MKR defined as the percentage of keypoints that are detected in the ground truth frame but not in the generated one. This metric assesses the appearance quality of each generated frame.
• Average Euclidean Distance (AED). Considering an externally trained image representation, we report the average euclidean distance between the ground truth and generated frame representation, similarly to Esser et al. [45]. We employ the feature embedding used in Monkey-Net [9].

Ablation Study. We compare the following variants of our model. Baseline: the simplest model trained without using the occlusion mask ($\mathcal{O}{\mathbf{S} \leftarrow \mathbf{D}}$ =1 in Equation 6), jacobians ($J_k= \mathbb{1}$ in Equation 4) and is supervised with $L{rec}$ at the highest resolution only; Pyr.: the pyramid loss is added to Baseline; Pyr.+ $\mathcal{O}_{\mathbf{S} \leftarrow \mathbf{D}}$: with respect to Pyr., we replace the generator network with the occlusion-aware network; Jac. w/o Equation 8 our model with local affine transformations but without equivariance constraints on jacobians Equation 8; Full: the full model including local affine transformations described in Section 3.1.

In Figure 3, we report the qualitative ablation. First, the pyramid loss leads to better results according to all the metrics except AKD. Second, adding $\mathcal{O}_{\mathbf{S} \leftarrow \mathbf{D}}$ to the model consistently improves all the metrics with respect to Pyr.. This illustrates the benefit of explicitly modeling occlusions. We found that without equivariance constraint over the jacobians, $J_k$ becomes unstable which leads to poor motion estimations. Finally, our Full model further improves all the metrics. In particular, we note that, with respect to the Baseline model, the MKR of the full model is smaller by the factor of 2.75. It shows that our rich motion representation helps generate more realistic images. These results are confirmed by our qualitative evaluation in Table 1 where we compare the Baseline and the Full models. In these experiments, each frame $\mathbf{D}$ of the input video is reconstructed from its first frame (first column) and the estimated keypoint trajectories. We note that the Baseline model does not locate any keypoints in the arms area. Consequently, when the pose difference with the initial pose increases, the model cannot reconstruct the video (columns 3, 4 and 5). In contrast, the Full model learns to detect a keypoint on each arm, and therefore, to more accurately reconstruct the input video even in the case of complex motion.

Comparison with State of the Art. We now compare our method with state of the art for the video reconstruction task as in [9]. To the best of our knowledge, X2Face [11] and Monkey-Net [9] are the only previous approaches for model-free image animation. Quantitative results are reported in Table 3. We observe that our approach consistently improves every single metric for each of the four different datasets. Even on the two face datasets, VoxCeleb and Nemo datasets, our approach clearly outperforms X2Face that was originally proposed for face generation. The better performance of our approach compared to X2Face is especially impressive X2Face exploits a larger motion embedding (128 floats) than our approach (60=K*(2+4) floats). Compared to Monkey-Net that uses a motion representation with a similar dimension (50=K*(2+3)), the advantages of our approach are clearly visible on the Tai-Chi-HD dataset that contains highly non-rigid objects (i.e. human body).

\begin{tabular}{c|ccc|ccc|ccc|c}
  \toprule
  & \multicolumn{3}{c|}{\emph{Tai-Chi-HD}} & \multicolumn{3}{c|}{\emph{VoxCeleb}} & \multicolumn{3}{c|}{\emph{Nemo}} & \multicolumn{1}{c}{\emph{Bair}} \\
  & $\mathcal{L}_1$ & ({AKD}, {MKR}) & {AED} & $\mathcal{L}_1$ & {AKD} & {AED} & $\mathcal{L}_1$ & {AKD} & {AED} & $\mathcal{L}_1$ \\ \toprule
  X2Face [11]& 0.080 & (17.654, 0.109) & 0.272 & 0.078 & 7.687 & 0.405 & 0.031 & 3.539 & 0.221 & 0.065 \\
  Monkey-Net [9] & 0.077 & (10.798, 0.059) & 0.228 & 0.049 & 1.878 & 0.199 & 0.018 & 1.285 & 0.077 & 0.034 \\
  Ours & \bf0.063 & (\bf6.862, \bf0.036) & \bf0.179 & \bf0.043 & \bf1.294 & \bf0.140 & \bf0.016 & \bf1.119 & \bf0.048 & \bf0.027\\
  \bottomrule
  \end{tabular}

We now report a qualitative comparison for image animation. Generated sequences are reported in Figure 4. The results are well in line with the quantitative evaluation in Table 3. Indeed, in both examples, X2Face and Monkey-Net are not able to correctly transfer the body notion in the driving video, instead warping the human body in the source image as a blob. Conversely, our approach is able to generate significantly better looking videos in which each body part is independently animated. This qualitative evaluation illustrates the potential of our rich motion description. We complete our evaluation with a user study. We ask users to select the most realistic image animation. Each question consists of the source image, the driving video, and the corresponding results of our method and a competitive method. We require each question to be answered by 10 AMT worker. This evaluation is repeated on 50 different input pairs. Results are reported in Table 2. We observe that our method is clearly preferred over the competitor methods. Interestingly, the largest difference with the state of the art is obtained on Tai-Chi-HD: the most challenging dataset in our evaluation due to its rich motions.

5. Conclusions

Section Summary: This study introduces a new method for animating images by tracking key points and applying small local adjustments to capture movement between frames, using a simplified math formula for quick calculations. A neural network then blends the look of an original image with the motion from a video, while also handling hidden or blocked areas by filling them in automatically. Tests show this approach works better than existing top methods across various benchmarks, both in numbers and visual quality.

We presented a novel approach for image animation based on keypoints and local affine transformations. Our novel mathematical formulation describes the motion field between two frames and is efficiently computed by deriving a first order Taylor expansion approximation. In this way, motion is described as a set of keypoints displacements and local affine transformations. A generator network combines the appearance of the source image and the motion representation of the driving video. In addition, we proposed to explicitly model occlusions in order to indicate to the generator network which image parts should be inpainted. We evaluated the proposed method both quantitatively and qualitatively and showed that our approach clearly outperforms state of the art on all the benchmarks.

A Detailed Derivations

Section Summary: This section derives a mathematical approximation for transforming motion between video frames near specific keypoints, using Taylor expansions to simplify complex transformations into local linear changes, and breaking them down through compositions of reference and driving frame mappings. It then explains equivariance constraints during training, which ensure that transformations between different frames remain consistent by matching positions and derivatives in their expansions, preventing distortions. Finally, it outlines a method to transfer only relative motion patterns by shifting movements from a driving video to align with keypoints in a source frame.

A.1 Approximating Motion with Local Affine Transformations

Here, we detail the derivation leading to the approximation of $\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{D}}$ near the keypoint $z_k$ in Equation 4. Using first order Taylor expansion we can obtain:

$ \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z) = \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z_k) + \left(\frac{d}{dz}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z) \middle\vert{z=z_k} \right)(z - z_k) + o(\Vert z - z_k \Vert)\tag{12} $

$\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{D}}$ can be written as the composition of two transformations:

$ \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}} = \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}} \circ \mathcal{T}_{\mathbf{R} \leftarrow \mathbf{D}}\tag{13} $

In order to compute the zeroth order term, we estimate the transformation $\mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}$ near the point $z_k$ in the driving frame $\mathbf{D}$, e.g $p_k = \mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}(z_k)$. Then we can estimate the transformation $\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}$ near $p_k$ in the reference $\mathbf{R}$. Since $p_k = \mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}(z_k)$ and $\mathcal{T}^{-1}{\mathbf{R} \leftarrow \mathbf{D}}=\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}$, we can write $z_k = \mathcal{T}_{\mathbf{D} \leftarrow \mathbf{R}}(p_k)$. Consequently, we obtain:

$ \begin{align} \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z_k) &= \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}} \circ \mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}(z_k)\notag \ &=\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}} \circ \mathcal{T}^{-1}{\mathbf{D} \leftarrow \mathbf{R}}(z_k)\notag\ &=\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}} \circ \mathcal{T}^{-1}{\mathbf{D} \leftarrow \mathbf{R}} \circ \mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p_k)\notag\ &=\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{R}}(p_k). \end{align}\tag{14} $

Concerning the first order term, we apply the function composition rule in Equation 13 and obtain:

$ \left(\frac{d}{dz}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z) \middle\vert{z=z_k} \right) = \left(\frac{d}{dp}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p) \middle\vert{p=\mathcal{T}{\mathbf{R} \leftarrow \mathbf{D}}(z_k)} \right) \left(\frac{d}{dz}\mathcal{T}^{-1}{\mathbf{D} \leftarrow \mathbf{R}}(z) \middle\vert_{z=z_k} \right)\tag{15} $

Since the matrix inverse of the Jacobian is equal to the Jacobian of the inverse function, and since $p_k = \mathcal{T}_{\mathbf{R} \leftarrow \mathbf{D}}(z_k)$, Equation 15 can be rewritten:

$ \left(\frac{d}{dz}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z) \middle\vert{z=z_k} \right) = \left(\frac{d}{dp}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)^{-1}\tag{16} $

After injecting Eqs. 14 and 16 into Equation 12, we finally obtain:

$ \mathcal{T}{\mathbf{S} \leftarrow \mathbf{D}}(z) \approx \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k) + \left(\frac{d}{dp}\mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{D} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)^{-1}(z - \mathcal{T}_{\mathbf{D} \leftarrow \mathbf{R}}(p_k))\tag{17} $

A.2 Equivariance Loss

At training time, we use equivariance constraints that enforces:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}} \equiv \mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}_{\mathbf{Y} \leftarrow \mathbf{R}}\tag{18} $

After applying first order Taylor expansion on the left-hand side, we obtain:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) = \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p_k) + \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)(p - p_k) + o(\Vert p - p_k \Vert). $

After applying first order Taylor expansion on the right-hand side in Equation 18, we obtain:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p) = \mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p_k) + \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}} \middle\vert_{p=p_k} \right)(p - p_k) + o(\Vert p - p_k \Vert), $

We can further simplify this expression using derivative of function composition:

$ \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}} \middle\vert_{p=p_k} \right) = \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}(p) \middle\vert{p=\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p_k)} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p) \middle\vert_{p=p_k} \right). $

Equation 18 holds only when every coefficient in Taylor expansion of the right and left sides are equal. Thus, it leads us to the following constaints:

$ \mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p_k) \equiv \mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}} \circ \mathcal{T}_{\mathbf{Y} \leftarrow \mathbf{R}} (p_k), $

and

$ \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \equiv \left(\frac{d}{dp}\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}(p) \middle\vert{p=\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p_k)} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{Y} \leftarrow \mathbf{R}}(p) \middle\vert_{p=p_k} \right). $

A.3 Transferring Relative Motion

In order to transfer only relative motion patterns, we propose to estimate $\mathcal{T}_{\mathbf{S}t \leftarrow \mathbf{R}}(p)$ near the keypoint $p_k$ by shifting the motion in the driving video to the location of keypoint $p_k$ in the source. To this aim, we introduce $\mathcal{V}{\mathbf{S}1 \leftarrow \mathbf{D_1}}(p_k)=\mathcal{T}{\mathbf{S}1 \leftarrow \mathbf{R}}(p_k)-\mathcal{T}{\mathbf{D}_1 \leftarrow \mathbf{R}}(p_k) \in \mathbb{R}^2$ that is the 2D vector from the landmark position $p_k$ in $\mathbf{D}_1$ to its position in $\mathbf{S}1$. We proceed as follows. First, we shift point coordinates according to $-\mathcal{V}{\mathbf{S}1 \leftarrow \mathbf{D_1}}(p_k)$ in order to obtain coordinates in $\mathbf{D_1}$. Second, we apply the transformation $\mathcal{T}{\mathbf{D}_t \leftarrow \mathbf{D}1}$. Finally, we translate the points back in the original coordinate space using $\mathcal{V}{\mathbf{S}_1 \leftarrow \mathbf{D_1}}(p_k)$. Formally, it can be written:

$ \mathcal{T}_{\mathbf{S}t \leftarrow \mathbf{R}}(p) = \mathcal{T}{\mathbf{D}_t \leftarrow \mathbf{D}1} \big(\mathcal{T}{\mathbf{S}1 \leftarrow \mathbf{R}}(p) - \mathcal{V}{\mathbf{S}_1 \leftarrow \mathbf{D}1}(p_k)\big) + \mathcal{V}{\mathbf{S}_1 \leftarrow \mathbf{D}_1}(p_k) $

Now, we can compute the value and Jacobian in the $p_k$:

$ \mathcal{T}_{\mathbf{S}t \leftarrow \mathbf{R}}(p_k) = \mathcal{T}{\mathbf{D}_t \leftarrow \mathbf{D}1} \circ \mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p_k) -\mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p_k) + \mathcal{T}{\mathbf{S}_1 \leftarrow \mathbf{R}}(p_k) $

and:

$ \left(\frac{d}{dp}\mathcal{T}{\mathbf{S}t \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) = \left(\frac{d}{dp}\mathcal{T}{\mathbf{D}t \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) ^ {-1} \left(\frac{d}{dp}\mathcal{T}{\mathbf{S}1 \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right). $

Now using Equation 17 and treating $\mathbf{S}_1$ as source and $\mathbf{S}_t$ as driving frame, we obtain:

$ \mathcal{T}_{\mathbf{S}1 \leftarrow \mathbf{S}t}(z) \approx \mathcal{T}{\mathbf{S}1 \leftarrow \mathbf{R}}(p_k) + J_k (z - \mathcal{T}{\mathbf{S} \leftarrow \mathbf{R}}(p_k) + \mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p_k) - \mathcal{T}{\mathbf{D}_t \leftarrow \mathbf{R}}(p_k))\tag{19} $

with

$ J_k=\left(\frac{d}{dp}\mathcal{T}{\mathbf{D}1 \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right) \left(\frac{d}{dp}\mathcal{T}{\mathbf{D}t \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)^{-1}.\tag{20} $

Note that, here, $\left(\frac{d}{dp}\mathcal{T}_{\mathbf{S}1 \leftarrow \mathbf{R}}(p) \middle\vert{p=p_k} \right)$ canceled out.

B Implementation details

Section Summary: To make the model more efficient by using less memory and computing power, the keypoint detector and motion predictor process images at a lower 64-by-64 resolution rather than the usual 256-by-256, while the motion module relies on a U-Net design with encoder and decoder blocks featuring convolutions, batch normalization, ReLU activations, pooling, and upsampling. The generator network follows a specific architecture with down-sampling, residual, and up-sampling blocks, and the whole system is trained using the Adam optimizer at a learning rate of 0.0002 with a batch size of 20, including scheduled rate reductions over about 40,000 to 100,000 iterations depending on the dataset, typically converging in two days on two high-end GPUs for larger sets. For the equivariance loss, which ensures the keypoint detector handles certain image transformations consistently, random thin-plate splines are applied with parameters drawn from normal distributions—small variances for deformation on a 5-by-5 grid and slightly larger for affine changes—to mimic realistic shifts.

B.1 Architecture details

In order to reduce memory and computational requirements of our model, the keypoint detector and dense motion predictor both work on resolution of $64 \times 64$ (instead of $256 \times 256$). For the two networks of the motion module, we employ an architecture based on U-Net [34] with five $conv_{3\times3}$ - $bn$ - $relu$ - $avg-pool_{2\times2}$ blocks in the encoders and five $upsample_{2\times2}$ - $conv_{3\times3}$ - $bn$ - $relu$ blocks in the decoders. In the generator network, we use the Johnson architecture [37] with two down-sampling blocks, six residual-blocks and two up-sampling blocks. We train our network using Adam [46] optimizer with learning rate $2e-4$ and batch size 20. We employ learning decay by dropping the learning rate at $\frac{T}{2}$ and $\frac{3T}{4}$ iterations, where T is total number of iteration. We chose $T \approx 100k$ for Tai-Chi-HD and VoxCeleb, and $T \approx 40k$ for Nemo and Bair. The model converges in approximately 2 days using 2 TitanX gpus for Tai-Chi-HD and VoxCeleb.

B.2 Equivariance loss implementation

As explained above our equivariance losses force the keypoint detector to be equivariant to some transformations $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}$. In our experiments $\mathcal{T}{\mathbf{X} \leftarrow \mathbf{Y}}$ is implemented using randomly sampled thin plate splines. We sample spline parameters from normal distributions with zero mean and variance equal to 0.005 for deformation component and 0.05 for the affine component. For deformation component we use uniform $5 \times 5$ grid.

C Additional experiments

Section Summary: This section presents extra experiments on image animation, comparing the method to prior techniques like X2face and Monkey-Net across datasets such as VoxCeleb, Nemo, Tai-Chi-HD, and Bair, where the new approach generates more realistic videos, better facial expressions, and accurate body movements without distorting backgrounds. It also visualizes self-learned keypoints, which consistently map to meaningful body parts or objects like faces, feet, or robot arms in the datasets. Finally, the predicted occlusion masks highlight hidden regions needing creative filling rather than simple image warping, performing reliably on human videos but adapting differently for robotic motion in the Bair dataset.

C.1 Image Animation

In this section, we report additional qualitative results.

We compare our approach with X2face [11] and Monkey-Net [9]. In Figure 5, we show three animation examples from the VoxCeleb dataset. First, X2face is not capable of generating realistic video sequences as we can see, for instance in the last frame of the last sequence. Then, Monkey-Net generates realistic frames but fails to generate specific facial expressions as in the third frame of the first sequence or in transferring the eye movements as in the last two frames of the second sequence.

In Figure 6, we show three animation examples from the Nemo dataset. First, we observe that this dataset is simpler than VoxCeleb since the persons are facing a uniformly black background. With this simpler dataset, X2Face generates realistic videos. However, it is not capable of inpainting image parts that are not visible in the source image. For instance, X2Face does not generate the teeth. Our approach also perform better than Monkey-Net as we can see by comparing the generate teeth in the first sequence or the closed eyes in the fourth frames of the second and third sequences.

In Figure 7, we report additional examples for the Tai-Chi-HD dataset. These examples are well in line with what is reported in the main paper. Both X2Face and Monkey-Net completely fail to generate realistic videos. The source images are warped without respecting human body structure. Conversely, our approach is able to deform the person in foreground without affecting the background. Even though we can see few minor artifacts, our model is able to move each body part independently following the body motion in the driving video.

Finally, in Figure 8 we show three image animation examples on the Bair dataset. Again, we see that X2Face is not able to transfer motion since it constantly returns frames almost identical with the source images. Compared to Monkey-Net, our approach performs slightly better since it preserves better the robot arm as we can see in the second frame of the first sequence or in the fourth frame of the last sequence.

C.2 Keypoint detection

We now illustrate the keypoints that are learned by our self-supervised approach in Figure 9. On the Tai-Chi-HD dataset, the keypoints are semantically consistent since each of them corresponds to a body part: light green for the right foot, and blue and red for the face for instance. Note that, a light green keypoint is constantly located in the bottom left corner in order to model background or camera motion. On VoxCeleb, we observe that, overall, the obtained keypoints are semantically consistent except for the yellow and green keypoints. For instance, the red and purple keypoints constantly correspond to the nose and the chin respectively. We observe a similar consistency for the Nemo dataset. For the Bair dataset, we note that two keypoints (dark blue and light green) correspond to the robotic arm.

C.3 Visualizing occlusion masks

In Figure 10, we visualize the predicted occlusion masks $\hat{\mathcal{O}}_{\mathbf{S}\leftarrow\mathbf{D}}$ on the Tai-Chi-HD, VoxCeleb and Nemo datasets. In the first sequence, when the person in the driving video is moving backward (second to fourth frames), the occlusion mask becomes black (corresponding to 0) in the background regions that are occluded in the source frame. It indicates that these parts cannot be generated by warping the source image features and must be inpainted. A similar observation can be made on the example sequence of VoxCeleb. Indeed, we see that when the face is rotating, the mask has low values (dark grey) in the neck region and in the right face side (in the left-hand side of the image) that are not visible in the source Frame. Then, since the driving video example from Nemo contains only little motion, the predicted mask is almost completely white. Overall, these three examples show that the occlusion masks truly indicate occluded regions even if no specific training loss is employed in order to lead to this behaviour. Finally, the predicted occlusion masks are more difficult to interpret in the case of the Bair dataset. Indeed, the robotic arm is masked out in every frame whereas we could expect that the model generates it by warping. A possible explanation is that, since in this particular dataset, the moving object is always the same, the network can generate without warping the source image. We observe also that masks have low values for the regions corresponding to the arm shadow. It is explained by the fact that shadows cannot be obtained by image warping and that they need to be added by the generator.

References

Section Summary: This references section compiles a bibliography of over 30 academic papers and articles, primarily from computer vision and graphics conferences between 1999 and 2019, that explore techniques for creating realistic digital images and videos of human faces, bodies, and movements. The works cover foundational methods like morphable 3D face models, real-time facial capture and reenactment, pose-guided animation, and generative adversarial networks (GANs) for synthesizing videos from images or audio. They also include studies on video prediction, landmark detection, and style transfer, providing key building blocks for AI tools that mimic human expressions and actions.

[1] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999.

[2] Chen Cao, Qiming Hou, and Kun Zhou. Displaced dynamic expression regression for real-time facial tracking and animation. TOG, 2014.

[3] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In CVPR, 2016.

[4] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag. Synthesizing images of humans in unseen poses. In CVPR, 2018.

[5] Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. In BMVC, 2019.

[6] Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan: Unsupervised video retargeting. In ECCV, 2018.

[7] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019.

[8] Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, Alexander Vakhitov, and Victor Lempitsky. Textured neural avatars. In CVPR, June 2019.

[9] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In CVPR, 2019.

[10] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In NIPS, 2018.

[11] Olivia Wiles, A Sophia Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In ECCV, 2018.

[12] Hao Tang, Wei Wang, Dan Xu, Yan Yan, and Nicu Sebe. Gesturegan for hand gesture-to-gesture translation in the wild. In ACM MM, 2018.

[13] Yahui Liu, Marco De Nadai, Gloria Zen, Nicu Sebe, and Bruno Lepri. Gesture-to-gesture translation in the wild via category-independent conditional maps. ACM MM, 2019.

[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.

[15] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.

[16] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In NIPS, 2018.

[17] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning to forecast and refine residual motion for image-to-video generation. In ECCV, 2018.

[18] A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: a large-scale speaker identification dataset. In INTERSPEECH, 2017.

[19] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NIPS, 2016.

[20] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In ICCV, 2017.

[21] Wei Wang, Xavier Alameda-Pineda, Dan Xu, Pascal Fua, Elisa Ricci, and Nicu Sebe. Every smile is unique: Landmark-guided diverse smile generation. In CVPR, 2018.

[22] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, 2018.

[23] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NIPS, 2016.

[24] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In NIPS, 2015.

[25] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In ICML, 2015.

[26] Joost Van Amersfoort, Anitha Kannan, Marc'Aurelio Ranzato, Arthur Szlam, Du Tran, and Soumith Chintala. Transformation-based models of video sequences. arXiv preprint arXiv:1701.08435, 2017.

[27] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2017.

[28] Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 3d guided fine-grained face manipulation. In CVPR, 2019.

[29] Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt. State of the art on monocular 3d face reconstruction, tracking, and applications. In Computer Graphics Forum, 2018.

[30] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ECCV, 2018.

[31] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.

[32] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, 2015.

[33] Joseph P Robinson, Yuncheng Li, Ning Zhang, Yun Fu, and Sergey Tulyakov. Laplace landmark localization. In ICCV, 2019.

[34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.

[35] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, and Nicu Sebe. Deformable gans for pose-based human image generation. In CVPR, 2018.

[36] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate-based texture inpainting for pose-guided image generation. In CVPR, 2019.

[37] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.

[38] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In ACSSC, 2003.

[39] Hao Tang, Dan Xu, Wei Wang, Yan Yan, and Nicu Sebe. Dual generator generative adversarial networks for multi-domain image-to-image translation. In ACCV, 2018.

[40] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised discovery of object landmarks as structural representations. In CVPR, 2018.

[41] Hamdi Dibeklioğlu, Albert Ali Salah, and Theo Gevers. Are you really smiling at me? spontaneous versus posed enjoyment smiles. In ECCV, 2012.

[42] Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, 2017.

[43] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In ICCV, 2017.

[44] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017.

[45] Patrick Esser, Ekaterina Sutter, and Björn Ommer. A variational u-net for conditional appearance and shape generation. In CVPR, 2018.

[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.