Down-Sampling Based Video Coding with Degradation-aware Restoration-Reconstruction Deep Neural Network (TIP’2021 - MMM’2020)

Man M. Ho, Gang He, Zheng Wang, and Jinjia Zhou.

News

Date News
2021/01/10 Our extended work has been accepted to and published in TIP’2021 (Open Access)
2020/07/05 We have updated the evaluation code for RR-DnCNN and RR-DnCNN v2.0
2020/01/12 Our RR-DnCNN will be published together with its version 2.0 at once
2020/01/07 We won Best Paper Runner-up Award at MMM’2020
2019/09/24 Our work has been accepted to MMM’2020 (Oral)

Papers:

Down-Sampling Based Video Coding with Degradation-aware Restoration-Reconstruction Deep Neural Network
Minh-Man Ho, Gang He, Zheng Wang, and Jinjia Zhou
In International Conference on Multimedia Modeling (MMM), 2020.

RR-DnCNN v2.0: Enhanced Restoration-Reconstruction Deep Neural Network for Down-Sampling Based Video Coding
(Open Access)
Man M. Ho, Jinjia Zhou, and Gang He
In IEEE Transactions on Image Processing (TIP), 2021.

Prerequisites

Getting Started.

Let’s reproduce our result.

1. Clone this repo:

git clone https://github.com/minhmanho/rrdncnn.git
cd rrdncnn

2. Prepare data:

Since the files are large, URLs here need your confirmation; otherwise, just run the command lines with recursion ‘-r’.

+ Johnny 1280x720 in Class E (65 MB) compressed using Low Delay P (recommended).

wget --no-check-certificate -r 'https://docs.google.com/uc?export=download&id=1PZWTdTkMxLpaSTJ4A7Xxw2QK6tmYy-d9' -O ./data/class_e.zip
unzip -o ./data/class_e.zip -d ./data/

or

./data/download_class_e.sh

+ BasketballDrill 832x480 in Class C (384 MB) compressed using All Intra (optional).

wget --no-check-certificate -r 'https://docs.google.com/uc?export=download&id=1GsnzTPGEVS8v-aMfQyrKExqKx-jZP7uP' -O ./data/class_c.zip
unzip -o ./data/class_c.zip -d ./data/

or

./data/download_class_c.sh

The expected data structure (for Johnny 1280x720 compressed by HEVC 16.20 with Low Delay P - LDP) is as:

./data/class_e
│   DHR_LDP.csv (HEVC on HR as <video_name>,<bir-rate kbps>, <Y-PSNR>, <U-PSNR>, <V-PSNR>, <YUV-PSNR>)
│   DLR_LDP.csv (HEVC on LR as <video_name>,<bir-rate kbps>, <Y-PSNR>, <U-PSNR>, <V-PSNR>, <YUV-PSNR>)
│   DHR_LDP_QP[32-47].txt (HEVC running logs)
|   DLR_LDP_QP[32-47].txt (HEVC running logs)
|
└───DLR_LDP_QP[32-47] (Decoded Low-Resolution)
│   │   Johnny_1280x720_50.yuv
|
└───HR (Uncompressed High-Resolution)
│   │   Johnny_1280x720_50.yuv
|
└───LR (Uncompressed Low-Resolution down-sampled x2 by bicubic)
│   │   Johnny_1280x720_50.yuv

3. Run models:

CUDA_VISIBLE_DEVICES=0 python eval.py --vin <path/to/class/folder> --ckpt <path/to/model> --size <video_size HxW> --decoder_folder <folder/contains/videos> --log_dir ./logs/

For example:

+ Run with RR-DnCNN.

CUDA_VISIBLE_DEVICES=0 python eval.py --vin ./data/class_e/ --ckpt ./models/rrdncnn.pth.tar --size 360x640 --decoder_folder DLR_LDP_QP32

+ Run with RR-DnCNN v2.0.

CUDA_VISIBLE_DEVICES=0 python eval.py --vin ./data/class_e/ --ckpt ./models/rrdncnn_v2.pth.tar --size 360x640 --decoder_folder DLR_LDP_QP32

+ Run with our prepared script.

./run.sh

Results

After running, you will get the results as follows:

+ <path/to/class/folder> or <args.out>/<model>_SDLR_<configuration>_<QP> (Folder contains reconstructed videos)
+ <args.log_dir>/<configuration>_<model>.csv (CSV file contains quantitative results)

Beside printing, we also export the results in CSV to log_dir for copying/pasting, where the columns (left-to-right) in the CSV file represent:

video_name_QP, HR-Y-PSNR, HR-Y-SSIM, LR-Y-PSNR, LR-Y-SSIM

E.g.,

rrdncnn_Johnny_1280x720_50_DLR_LDP_QP32,35.0897,0.9118,38.2423,0.9594

Finally, we calculate the BD-rates based on the results of HEVC (attached in the class package) and our methods. The expected output on Johnny (LDP) and BasketballDrill (AI) should be:

Sequence_QP HEVC 16.20 Low Bit-rate (kbps) RR-DnCNN RR-DnCNN v2.0
Bit-rate (kbps) PSNR (dB) PSNR BD-BR (%) PSNR (dB) BD-BR (%)
Johnny_1280x720_50.yuv_32 87.7059 38.9999 34.4353 35.0897 -12.9307 35.4468 -15.8121
Johnny_1280x720_50.yuv_37 46.9529 36.7226 18.9812 33.4981 33.6468
Johnny_1280x720_50.yuv_42 25.5059 34.1018 11.4706 31.2232 31.3090
Johnny_1280x720_50.yuv_47 14.9129 31.5612 7.0847 28.7739 28.8354
BasketballDrill_832x480_50.yuv_32 1833.6642 35.4191 685.0295 31.0428 -10.9197 31.0627 -12.4616
BasketballDrill_832x480_50.yuv_37 1018.0177 32.7758 392.1459 29.2932 29.365
BasketballDrill_832x480_50.yuv_42 559.657 30.1624 208.8544 26.9593 27.0212
BasketballDrill_832x480_50.yuv_47 277.0484 27.5161 97.052 24.7342 24.7411

Let’s check our pre-run version (26 KB) for more details.

wget 'https://docs.google.com/uc?export=download&id=1VkiMalZp_UxejxA3ScoE7fzDDZoWACFc' -O ./logs/result_sample.zip
unzip -o ./logs/result_sample.zip -d ./logs/

or

./logs/download_result_sample.sh

Citations

Please cite this work if you find it useful.

Version 1.0:

@inproceedings{ho2020down,
  title={Down-sampling based video coding with degradation-aware restoration-reconstruction deep neural network},
  author={Ho, Minh-Man and He, Gang and Wang, Zheng and Zhou, Jinjia},
  booktitle={International Conference on Multimedia Modeling},
  pages={99--110},
  year={2020},
  organization={Springer}
}

Version 2.0:

@article{ho2021rr,
  title={RR-DnCNN v2.0: Enhanced Restoration-Reconstruction Deep Neural Network for Down-Sampling-Based Video Coding},
  author={Ho, Man M and Zhou, Jinjia and He, Gang},
  journal={IEEE Transactions on Image Processing},
  volume={30},
  pages={1702--1715},
  year={2021},
  publisher={IEEE}
}

License

This repository (as well as its materials) is for non-commercial uses and research purposes only.

Acknowledgement

Thank G. Bjontegaard and S. Pateux for ETRO’s Bjontegaard Metric Implementation.

G. Bjontegaard, Calculation of average PSNR differences between RD-curves (VCEG-M33)
S. Pateux, J. Jung, An excel add-in for computing Bjontegaard metric and its evolution

This work is supported by JST, PRESTO Grant Number JPMJPR1757 Japan.

Contact

If you have any questions, or the use of material violates your copyright / license, please contact me manminhho.cs@gmail.com.