/** **/

ITCooky Recipes

Lets cooky it yammy things!

How to rejuvenate, change the gender of the face in the video with the STIT neural network – Stitch it in Time

дата April 17, 2022

Another video effects project with neuroprocessing STIT – Stitch it in Time can rejuvenate the face, age, change sex and much more, but with a but!

First the disadvantages noted:
– Poorly described, nothing at all except a couple of examples.
– It doesn’t make a full video with sound, it will show some kind of demo without sound (can be fixed)
– Cannot choose a face if there are many in the frame (cannot be corrected)
– You can’t select only a face. exmaple if you put your hand on it

There are no necessary features, and of those that do exist, some are poorly done I suspect on purpose, so this project is limitedly applicable to podcasters and other hosts who don’t scratch their heads.

I take Ubuntu and an Nvidia video card with CUDA
My PC:
OS: Ubuntu 20.04.3 LTS
CPU: AMD® Ryzen 7 3700x 8-core processor × 16
MB: Gigabyte X570 I AORUS PRO WIFI
Mem: 64 GB DDR4 1330 Mhz
GPU: Nvidia GeForce RTX 2070 8Gb (2304 CUDA CORES)
Disk: Samsung SSD 512Gb EVO Plus

A and we also take the Spanish host Arturo Valls!

You can of course take Windows and work with this project via WSL2 as in this article Resize and enhance old photos with GFPGAN on Windows via WSL2 and a card with CUDA (or without)!, but consider one thing, there are a lot of calculations here, and I noticed that CUDA on Windows via WSL2 works 20 times slower than on Linux

Install CUDA
In the Ubuntu console, run, and see the Nvidia driver
nvidia-smi

Sun Apr  3 14:53:22 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:0A:00.0  On |                  N/A |
| 90%   73C    P8    27W / 175W |    409MiB /  7979MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1736      G   /usr/lib/xorg/Xorg                 35MiB |
|    0   N/A  N/A      2362      G   /usr/lib/xorg/Xorg                142MiB |
|    0   N/A  N/A      2476      G   /usr/bin/gnome-shell               43MiB |
|    0   N/A  N/A     59650      G   /usr/lib/firefox/firefox          155MiB |
+-----------------------------------------------------------------------------+

NVIDIA driver is already installed, if not, please install it yourself

Upgrade
sudo apt-get update && sudo apt-get upgrade -y

We install CUDA, I already have everything I need installed, but generally this is done like this
sudo apt install nvidia-cuda-toolkit

Install Conda
Python has a bug or a feature: what is written in one version may not work in others, the same applies to different versions of modules. And conda creates an environment for each program with the necessary versions and no update will break them.

Downloading a minimalist version of Conda
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh

We answer yes to all questions during installation.

You have to close open the Ubuntu console run
conda config --show
Something shows that means it works

Install STIT
Creating a Python environment for GFPGAN version 3.7

conda create --name STIT python=3.7
conda activate STIT

We do what the website creators do
git clone https://github.com/rotemtzaban/STIT.git
cd STIT
pip install -r requirements.txt

They say they tested it on tourch version 1.10 and it will installed today
pip install torch torchvision
pip install git+https://github.com/openai/CLIP.git

Download modelos
pip install gdown
bash ./download_models.sh

Lets start processing
I make a folder in STIT
mkdir arturo
Here I shot frames from the video file data.mp4
ffmpeg -i "data.mp4" "arturo/out%04d.png"

I create another folder
mkdir ./arturo/experiment_dir
And there I start to create a model… very similar to a face model…
python train.py --input_folder ./arturo --output_folder ./arturo/experiment_dir --run_name arturo --num_pti_steps 80
Here you have to change
–input_folder – folder ./arturo where before we put frames of the video on which we will superimpose
–output_folder – folder ./arturo/experiment_dir of model
–run_name – just name arturo
Here we can change
–num_pti_steps – this is the number of iterations, it affects the execution time of this command and presumably the clarity of the result

Next, I create a folder where the output will be thrown.
mkdir arturo/edits

Lets rejuvenate the face in the STIT video
Effect age settings -4 -4 1

Next, I run the rejuvenation command, it contains folders for the model ./arthur and indicated where to put ./arturo/edits
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name age --edit_range -4 -4 1 --outer_mask_dilation 50
Here you have to change
–input_folder – folder ./arturo where we put the frames of the video on which we are going to superimpose
–output_folder – folder ./arturo/experiment_dir where does the model come from
–run_name – a name just a name arturo
–edit_name – effect to apply age
–edit_range – effect settings -4 -4 1 means it will rejuvenate in 4 levels
–outer_mask_dilation – found in examples 50 i put 1 and everything turned black instead of the face

By the way, after examining the contents of the /STIT/editings/w_directions folder you can get an idea of the available effects by name

age.npy                   eyes_open.npy   mouth_ratio.npy          pitch.npy
eye_distance.npy          gender.npy      nose_mouth_distance.npy  roll.npy
eye_eyebrow_distance.npy  lip_ratio.npy   nose_ratio.npy           smile.npy
eye_ratio.npy             mouth_open.npy  nose_tip.npy             yaw.npy

My video is 1920×1080 and in a folder /STIT/arturo/edits/age/-4.0 appears video out.mp4 of size 5760×1344 like this

Why do they do it like that…

I cut the desired piece
ffmpeg -i out.mp4 -filter:v "crop=1920:1080:3840:264" -c:a copy young.mp4

Turns out STIT changed the duration of the video.
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 young.mp4
3.040000 but it must be 2.240000

fixing it
ffmpeg -i younge.mp4 -filter:v "setpts=(2.24/3.04)*PTS" younge2.mp4
Video still broken 2.32 instead of 2.24 but not as noticeable!

For comparison, join it with the original
ffmpeg -i data.mp4 -i young2.mp4 -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" out_comp.mp4
I make a screenshot – obvious rejuvenation

By the way, the duration of the video is broken on purpose!!!

I change in files edit_video.py and edit_video_stitching_tuning.py online

imageio.mimwrite(os.path.join(folder_path, 'out.mp4'), frames, fps=18, output_params=['-vf', 'fps=25'])

18 to 25

imageio.mimwrite(os.path.join(folder_path, 'out.mp4'), frames, fps=25, output_params=['-vf', 'fps=25'])

I ran the effect render again and the video length turned out to be 2.2, just 0.04 shorter than it was.

I’ll go back to model creation, increase –num_pti_steps to 420
python train.py --input_folder ./arturo --output_folder ./arturo/experiment_dir --run_name arturo --num_pti_steps 420
Processing 55 frames (2.2 seconds of video) takes 70 minutes on my PC

And I will start the rejuvenation again, I will try to feel the difference in the image, the process takes 16 minutes.

Here and below I compare doing
I cut the original video from the STIT result so that the time match is total
ffmpeg -i out.mp4 -filter:v "crop=1920:1080:0:264" -c:a copy src.mp4
I cut a piece of it with a video face
ffmpeg -i src.mp4 -filter:v "crop=860:1080:700:0" -c:a copy src0.mp4
I cut out a part with an effect of the STIT result
ffmpeg -i out.mp4 -filter:v "crop=1920:1080:3840:264" -c:a copy young.mp4
I cut a piece of it with a video face
ffmpeg -i young.mp4 -filter:v "crop=860:1080:700:0" -c:a copy young0.mp4
join it, original on the left
ffmpeg -i src0.mp4 -i young0.mp4 -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" young_comp.mp4
and i take a screenshot for comparison

There is not much difference, but if you look closely at the teeth, you can see that it is a little lighter and the wrinkles on the forehead are lighter.

Rejuvenate the face even more in the video with STIT
Effect age settings -8 -8 1
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name age --edit_range -8 -8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Parameter added here
–border_loss_threshold – As far as I understand, this affects the end of the processing of some sections and more resources dedicated to others, for videos with a simple background they put 0.005 with complex 0.002
–outer_mask_dilation – reduced here to 25

Old face on video with STIT
Effect age settings 8 8 1
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name age --edit_range 8 8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Open the eyes to the face in video with STIT
Effect eyes_open settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eyes_open --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Weakly noticeable effect at 32

And at 128

Close the eyes to the face in video with STIT
Effect eye_ratio settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_ratio --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Again open your eyes to the video with STIT
Effect eye_ratio settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_ratio --edit_range -32 -32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Cover the mouth on video with STIT
Effect mouth_ratio settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_ratio --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Opening the mouth wider on video with STIT
Effect mouth_ratio settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_ratio --edit_range -32 -32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Opening the mouth wider again on video with STIT
Also an effect in the mouth mouth_open settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_open --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Close the mouth with STIT on video
Also an effect in the mouth mouth_open settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_open --edit_range -32 -32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Replace to a face with masculine characteristics the original face in a video with STIT
Effect gender settings 8 8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name gender --edit_range 8 8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Replace to a face with female characteristics the original face in a video with STIT
Effect gender settings -8 -8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name gender --edit_range -8 -8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Surprisingly, Arturo became very similar to Silvia Abril, once upon a time she replaced him in Ahora Caigo.

Big smile on the face on video with STIT
Effect smile settings 4 4
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name smile --edit_range 4 4 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Tense smile on the face in video with STIT
Effect smile settings -4 -4
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name smile --edit_range -4 -4 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Flat nose on video with STIT
Effect pitch settings 8 8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name pitch --edit_range 8 8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Reverse flat nose on video with STIT
Effect pitch settings -8 -8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name pitch --edit_range -8 -8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Increasing the distance between the nose and mouth on the face in the video with STIT
Effect nose_mouth_distance settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name nose_mouth_distance --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Poor visible but perceptible

Decrease in the distance between the nose and the mouth in the face in the video with STIT
Effect nose_mouth_distance settings -48 -48
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name nose_mouth_distance --edit_range -48 -48 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

Cleverly reduces the distance between the nose and the lip by exposing the teeth

The distance of the eyes on the face in the video with STIT
Effect eye_distance settings 64 64
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_distance --edit_range 64 64 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

I think this filter does not do what it says

Eye distance with a – on the face in the video with STIT
Effect eye_distance settings -64 -64
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_distance --edit_range -64 -64 1 --outer_mask_dilation 25 --border_loss_threshold 0.005

I think this filter does not do what it says

I also tried eye_eyebrow_distance, roll, lip_ratio, yaw, nose_ratio, nose_tip effects with values -128 -64 -32 -8 8 32 64 128 I did not notice any special effects (in some cases there was some deffects)!

All STIT effects in one video

Conclusion: STIT can be used to add age, gender, smile effects to one face videos. I think it adds some deffects to the video as well; for example, I find that it makes the face slimmer. And the effects are not clean: for example, you make a woman gender, and STIT opens the mouth too!


Leave a Reply

Your email address will not be published.