Another video effects project with neuroprocessing STIT – Stitch it in Time can rejuvenate the face, age, change sex and much more, but with a but!
First the disadvantages noted:
– Poorly described, nothing at all except a couple of examples.
– It doesn’t make a full video with sound, it will show some kind of demo without sound (can be fixed)
– Cannot choose a face if there are many in the frame (cannot be corrected)
– You can’t select only a face. exmaple if you put your hand on it
There are no necessary features, and of those that do exist, some are poorly done I suspect on purpose, so this project is limitedly applicable to podcasters and other hosts who don’t scratch their heads.
I take Ubuntu and an Nvidia video card with CUDA
My PC:
OS: Ubuntu 20.04.3 LTS
CPU: AMD® Ryzen 7 3700x 8-core processor × 16
MB: Gigabyte X570 I AORUS PRO WIFI
Mem: 64 GB DDR4 1330 Mhz
GPU: Nvidia GeForce RTX 2070 8Gb (2304 CUDA CORES)
Disk: Samsung SSD 512Gb EVO Plus
A and we also take the Spanish host Arturo Valls!
You can of course take Windows and work with this project via WSL2 as in this article Resize and enhance old photos with GFPGAN on Windows via WSL2 and a card with CUDA (or without)!, but consider one thing, there are a lot of calculations here, and I noticed that CUDA on Windows via WSL2 works 20 times slower than on Linux
Install CUDA
In the Ubuntu console, run, and see the Nvidia driver
nvidia-smi
Sun Apr 3 14:53:22 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:0A:00.0 On | N/A |
| 90% 73C P8 27W / 175W | 409MiB / 7979MiB | 7% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1736 G /usr/lib/xorg/Xorg 35MiB |
| 0 N/A N/A 2362 G /usr/lib/xorg/Xorg 142MiB |
| 0 N/A N/A 2476 G /usr/bin/gnome-shell 43MiB |
| 0 N/A N/A 59650 G /usr/lib/firefox/firefox 155MiB |
+-----------------------------------------------------------------------------+
NVIDIA driver is already installed, if not, please install it yourself
Upgrade
sudo apt-get update && sudo apt-get upgrade -y
We install CUDA, I already have everything I need installed, but generally this is done like this
sudo apt install nvidia-cuda-toolkit
Install Conda
Python has a bug or a feature: what is written in one version may not work in others, the same applies to different versions of modules. And conda creates an environment for each program with the necessary versions and no update will break them.
Downloading a minimalist version of Conda
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh
We answer yes to all questions during installation.
You have to close open the Ubuntu console run
conda config --show
Something shows that means it works
Install STIT
Creating a Python environment for GFPGAN version 3.7
conda create --name STIT python=3.7
conda activate STIT
We do what the website creators do
git clone https://github.com/rotemtzaban/STIT.git
cd STIT
pip install -r requirements.txt
They say they tested it on tourch version 1.10 and it will installed today
pip install torch torchvision
pip install git+https://github.com/openai/CLIP.git
Download modelos
pip install gdown
bash ./download_models.sh
Lets start processing
I make a folder in STIT
mkdir arturo
Here I shot frames from the video file data.mp4
ffmpeg -i "data.mp4" "arturo/out%04d.png"
I create another folder
mkdir ./arturo/experiment_dir
And there I start to create a model… very similar to a face model…
python train.py --input_folder ./arturo --output_folder ./arturo/experiment_dir --run_name arturo --num_pti_steps 80
Here you have to change
–input_folder – folder ./arturo where before we put frames of the video on which we will superimpose
–output_folder – folder ./arturo/experiment_dir of model
–run_name – just name arturo
Here we can change
–num_pti_steps – this is the number of iterations, it affects the execution time of this command and presumably the clarity of the result
Next, I create a folder where the output will be thrown.
mkdir arturo/edits
Lets rejuvenate the face in the STIT video
Effect age settings -4 -4 1
Next, I run the rejuvenation command, it contains folders for the model ./arthur and indicated where to put ./arturo/edits
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name age --edit_range -4 -4 1 --outer_mask_dilation 50
Here you have to change
–input_folder – folder ./arturo where we put the frames of the video on which we are going to superimpose
–output_folder – folder ./arturo/experiment_dir where does the model come from
–run_name – a name just a name arturo
–edit_name – effect to apply age
–edit_range – effect settings -4 -4 1 means it will rejuvenate in 4 levels
–outer_mask_dilation – found in examples 50 i put 1 and everything turned black instead of the face
By the way, after examining the contents of the /STIT/editings/w_directions folder you can get an idea of the available effects by name
age.npy eyes_open.npy mouth_ratio.npy pitch.npy eye_distance.npy gender.npy nose_mouth_distance.npy roll.npy eye_eyebrow_distance.npy lip_ratio.npy nose_ratio.npy smile.npy eye_ratio.npy mouth_open.npy nose_tip.npy yaw.npy
My video is 1920×1080 and in a folder /STIT/arturo/edits/age/-4.0 appears video out.mp4 of size 5760×1344 like this
Why do they do it like that…
I cut the desired piece
ffmpeg -i out.mp4 -filter:v "crop=1920:1080:3840:264" -c:a copy young.mp4
Turns out STIT changed the duration of the video.
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 young.mp4
3.040000 but it must be 2.240000
fixing it
ffmpeg -i younge.mp4 -filter:v "setpts=(2.24/3.04)*PTS" younge2.mp4
Video still broken 2.32 instead of 2.24 but not as noticeable!
For comparison, join it with the original
ffmpeg -i data.mp4 -i young2.mp4 -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" out_comp.mp4
I make a screenshot – obvious rejuvenation
By the way, the duration of the video is broken on purpose!!!
I change in files edit_video.py and edit_video_stitching_tuning.py online
imageio.mimwrite(os.path.join(folder_path, 'out.mp4'), frames, fps=18, output_params=['-vf', 'fps=25'])
18 to 25
imageio.mimwrite(os.path.join(folder_path, 'out.mp4'), frames, fps=25, output_params=['-vf', 'fps=25'])
I ran the effect render again and the video length turned out to be 2.2, just 0.04 shorter than it was.
I’ll go back to model creation, increase –num_pti_steps to 420
python train.py --input_folder ./arturo --output_folder ./arturo/experiment_dir --run_name arturo --num_pti_steps 420
Processing 55 frames (2.2 seconds of video) takes 70 minutes on my PC
And I will start the rejuvenation again, I will try to feel the difference in the image, the process takes 16 minutes.
Here and below I compare doing
I cut the original video from the STIT result so that the time match is total
ffmpeg -i out.mp4 -filter:v "crop=1920:1080:0:264" -c:a copy src.mp4
I cut a piece of it with a video face
ffmpeg -i src.mp4 -filter:v "crop=860:1080:700:0" -c:a copy src0.mp4
I cut out a part with an effect of the STIT result
ffmpeg -i out.mp4 -filter:v "crop=1920:1080:3840:264" -c:a copy young.mp4
I cut a piece of it with a video face
ffmpeg -i young.mp4 -filter:v "crop=860:1080:700:0" -c:a copy young0.mp4
join it, original on the left
ffmpeg -i src0.mp4 -i young0.mp4 -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" young_comp.mp4
and i take a screenshot for comparison
There is not much difference, but if you look closely at the teeth, you can see that it is a little lighter and the wrinkles on the forehead are lighter.
Rejuvenate the face even more in the video with STIT
Effect age settings -8 -8 1
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name age --edit_range -8 -8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Parameter added here
–border_loss_threshold – As far as I understand, this affects the end of the processing of some sections and more resources dedicated to others, for videos with a simple background they put 0.005 with complex 0.002
–outer_mask_dilation – reduced here to 25
Old face on video with STIT
Effect age settings 8 8 1
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name age --edit_range 8 8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Open the eyes to the face in video with STIT
Effect eyes_open settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eyes_open --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Weakly noticeable effect at 32
And at 128
Close the eyes to the face in video with STIT
Effect eye_ratio settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_ratio --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Again open your eyes to the video with STIT
Effect eye_ratio settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_ratio --edit_range -32 -32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Cover the mouth on video with STIT
Effect mouth_ratio settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_ratio --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Opening the mouth wider on video with STIT
Effect mouth_ratio settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_ratio --edit_range -32 -32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Opening the mouth wider again on video with STIT
Also an effect in the mouth mouth_open settings 32 32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_open --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Close the mouth with STIT on video
Also an effect in the mouth mouth_open settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name mouth_open --edit_range -32 -32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Replace to a face with masculine characteristics the original face in a video with STIT
Effect gender settings 8 8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name gender --edit_range 8 8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Replace to a face with female characteristics the original face in a video with STIT
Effect gender settings -8 -8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name gender --edit_range -8 -8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Surprisingly, Arturo became very similar to Silvia Abril, once upon a time she replaced him in Ahora Caigo.
Big smile on the face on video with STIT
Effect smile settings 4 4
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name smile --edit_range 4 4 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Tense smile on the face in video with STIT
Effect smile settings -4 -4
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name smile --edit_range -4 -4 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Flat nose on video with STIT
Effect pitch settings 8 8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name pitch --edit_range 8 8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Reverse flat nose on video with STIT
Effect pitch settings -8 -8
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name pitch --edit_range -8 -8 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Increasing the distance between the nose and mouth on the face in the video with STIT
Effect nose_mouth_distance settings -32 -32
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name nose_mouth_distance --edit_range 32 32 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Poor visible but perceptible
Decrease in the distance between the nose and the mouth in the face in the video with STIT
Effect nose_mouth_distance settings -48 -48
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name nose_mouth_distance --edit_range -48 -48 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
Cleverly reduces the distance between the nose and the lip by exposing the teeth
The distance of the eyes on the face in the video with STIT
Effect eye_distance settings 64 64
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_distance --edit_range 64 64 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
I think this filter does not do what it says
Eye distance with a – on the face in the video with STIT
Effect eye_distance settings -64 -64
python edit_video_stitching_tuning.py --input_folder ./arturo --output_folder ./arturo/edits --run_name arturo --edit_name eye_distance --edit_range -64 -64 1 --outer_mask_dilation 25 --border_loss_threshold 0.005
I think this filter does not do what it says
I also tried eye_eyebrow_distance, roll, lip_ratio, yaw, nose_ratio, nose_tip effects with values -128 -64 -32 -8 8 32 64 128 I did not notice any special effects (in some cases there was some deffects)!
All STIT effects in one video
Conclusion: STIT can be used to add age, gender, smile effects to one face videos. I think it adds some deffects to the video as well; for example, I find that it makes the face slimmer. And the effects are not clean: for example, you make a woman gender, and STIT opens the mouth too!
Leave a Reply