Video Super-Resolution Quantization Work Log

Video Super-Resolution Quantization (Time:2023.07.07-2023.08.07)

Paper Reading

  1. Dynamic Network Quantization for Efficient Video Inference (ICCV2021)
    • Feat: selects optimal precision for each frame conditioned on the input for efficient video recognition
  2. ResQ: Residual Quantization for Video Perception (ICCV2023)
    • Feat: difference in network activations between two neighboring frames, exhibit properties that make them highly quantizable
  3. QuantSR: Accurate Low-bit Quantization for Efficient Image Super-Resolution (NIPS2023)
    • To overcome the representation homogeneity caused by quantization in the network, we introduce the Redistribution-driven Learnable Quantizer (RLQ). This is accomplished through an inference-agnostic efficient redistribution design, which adds additional information in both forward and backward passes to improve the representation ability of quantized networks. (为了克服网络中量化造成的表示同质性,我们引入了重分布驱动的可学习量化器 (RLQ)。这是通过与推理无关的高效重分布设计实现的,它在前向和后向传递中添加了额外信息,以提高量化网络的表示能力。)
    • Furthermore, to achieve flexible inference and break the upper limit of accuracy, we propose the Depth-dynamic Quantized Architecture (DQA). Our DQA allows for the trade-off between efficiency and accuracy during inference through weight sharing.(此外,为了实现灵活的推理并突破准确率的上限,我们提出了深度动态量化架构(DQA)。我们的DQA通过权重共享,实现了推理过程中效率和准确率之间的平衡。)
  4. Knowledge Distillation for Optical Flow-Based Video Superresolution (JCSE2023)
    • Feat: Video super-resolution; Optical flow; Knowledge distillation;
  5. EDVR: Video Restoration with Enhanced Deformable Convolutional Networks (NTIRE2019)
  6. leverage temporal redundancies to accelerate video processing
    1. Towards High Performance Video Object Detection for Mobiles (MSRA_arxiv2018)
    2. Temporally Distributed Networks for Fast Video Semantic Segmentation (CVPR2020)
      • Feat: 在连续帧上用前层网络获取浅层特征,通过 attention 将当前帧前的浅层特征传播到当前帧来近似得到在当前帧上使用深层网络获取深层特征的效果。在分割任务上简单高效
    3. Mobile Video Object Detection with Temporally-Aware Feature Maps (CVPR2018)
      • Feat: 来自之前帧的 hidden state 当作 temperal information 增强当前帧的目标检测效果
    4. Low-Latency Video Semantic Segmentation (CVPR2018)
      • Feat: 视频语义分割 当前帧处理受之前帧中间特征影响,判断是否为关键帧,关键帧用高计算强度的模块处理

Idea

  1. 需要搞清楚 basicvsr++ 模型接受的输入是怎样的,模型的大致处理过程是怎样的? input example: torch.Size([1, 141, 3, 240, 320]) -> finish
  2. 需要搞清楚 test 加载数据计算指标的 pipeline? 成功, test 结果如下: -> finish
    1. orig: 07/11 20:25:27 - mmengine - INFO - Iter(test) [4/4] REDS4-BIx4-RGB/PSNR: 32.3965 REDS4-BIx4-RGB/SSIM: 0.9075 data_time: 13.1019 time: 57.3645
    2. curret_best: 07/12 17:36:58 - mmengine - INFO - Iter(test) [4/4] REDS4-BIx4-RGB/PSNR: 25.3909 REDS4-BIx4-RGB/SSIM: 0.6822 data_time: 12.9891 time: 64.2525
    3. current_now: 07/12 22:35:48 - mmengine - INFO - Iter(test) [4/4] REDS4-BIx4-RGB/PSNR: 25.3899 REDS4-BIx4-RGB/SSIM: 0.6821 data_time: 13.4293 time: 71.2697
    4. current_all:
      1. REDS4-BIx4-RGB/PSNR: 25.3908 REDS4-BIx4-RGB/SSIM: 0.6821
      2. Vimeo-90K-T-BDx4-Y/PSNR: 29.6901 Vimeo-90K-T-BDx4-Y/SSIM: 0.8333 Vimeo-90K-T-BIx4-Y/PSNR: 30.3137 Vimeo-90K-T-BIx4-Y/SSIM: 0.8437
      3. UDM10-BDx4-Y/PSNR: 30.7291 UDM10-BDx4-Y/SSIM: 0.8677
      4. VID4-BDx4-Y/PSNR: 22.9580 VID4-BDx4-Y/SSIM: 0.5820 VID4-BIx4-Y/PSNR: 23.2985 VID4-BIx4-Y/SSIM: 0.5998
  3. 如何降低量化时间,提升量化后效果? -> cease
    1. current: the calibration time is 16175.18998336792 s 约 4.5 h) 暂时无解
  4. EDVR 在 REDS 上测试? -> cease
    1. orig_0: 07/16 16:04:32 - mmengine - INFO - Iter(test) [400/400] REDS4-BIx4-RGB/PSNR: 24.7137 SSIM: 0.6305 data_time: 0.1508 time: 0.5635
    2. orig_1: 07/16 16:15:56 - mmengine - INFO - Iter(test) [400/400] REDS4-BIx4-RGB/PSNR: 23.5544 SSIM: 0.6249 data_time: 0.1505 time: 0.5589
    3. orig_2: 07/16 18:42:34 - mmengine - INFO - Iter(test) [400/400] REDS4-BIx4-RGB/PSNR: 23.8858 REDS4-BIx4-RGB/SSIM: 0.6057 data_time: 0.1552 time: 0.5929
  5. 转向在 VSR 小模型上测试量化算法的效果 -> cease
    1. 小的视频超分模型几乎都有各自的特点:有用剪枝的 有突出功耗低的 有用重参数化技巧的 种种已有特点不适合再叠加量化算法
  6. 转向在 SISR 模型上测试量化算法的效果 -> cease
  7. 尝试其它轻量化技巧,聚焦移动设备应用
    1. 结构重参数

Metrics

  1. PSNR
  2. SSIM
  3. Memory
  4. Latency

Results

Milestone_0

Rank Model Source Dataset PSNR SSIM Memory Latency

Milestone_1

PaperWriting

No.1

PaperReference