Top: Compared with previous methods, our approach takes into account multiple complex distortions..
Recent advances in neural camera imaging pipelines have demonstrated notable progress. Nevertheless, the real-world imaging pipeline still faces challenges including the lack of joint optimization in system components, computational redundancies, and optical distortions such as lens shading. In light of this, we propose an end-to-end camera imaging pipeline (RealCamNet) to enhance real-world camera imaging performance. Our methodology diverges from conventional, fragmented multi-stage image signal processing towards end-to-end architecture. This architecture facilitates joint optimization across the full pipeline and the restoration of coordinate-biased distortions. RealCamNet is designed for high-quality conversion from RAW to RGB and compact image compression. Specifically, we deeply analyze coordinate-dependent optical distortions, e.g., vignetting and dark shading, and design a novel Coordinate-Aware Distortion Restoration (CADR) module to restore coordinate-biased distortions. Furthermore, we propose a Coordinate-Independent Mapping Compression (CIMC) module to implement tone mapping and redundant information compression. Existing datasets suffer from misalignment and overly idealized conditions, making them inadequate for training real-world imaging pipelines. Therefore, we collected a real-world imaging dataset. Experiment results show that RealCamNet achieves the best rate-distortion performance with lower inference latency.
Top: Examples of coordinate-related distortion (vignetting and dark shading).
Top: The architecture of RealCamNet.
Quantitative comparison on Rate-Distortion Curve.
Quantitative results. We compare with state-of-the-art ISPNet and image compression methods, including learning-based methods: PyNet(CVPR'20), LiteISPNet(ICCV'21), MwISPNet(ECCV'20), MLIC(ACMMM'23), TCM(CVPR'23) and the most advanced traditional image compression method VTM/H.266. We show BD-Rate, BD-PSNR, BD-MSSSIM, BD-δE and BD-LPPHS for all methods. We use PyNet+VTM as anchor.
Compared with the state-of-the-art methods, our framework, optimized end-to-end, demonstrates significant enhancements in imaging systems' performance. It offers comprehensive benefits, encompassing reduced bit rate, augmented color fidelity, and elevated PSNR.
We would like to thank Ma Zijia for his important contribution to the experiment, Xu Li for her constructive suggestions on the thesis design, and He Gang for his support in the process of establishing the thesis topic.
RealCamNet: An End-to-End Real-World Camera Imaging Pipeline
@inproceedings{realcamnet,
author = {Kepeng Xu and Zijia Ma and Li Xu and Gang He and Yunsong Li and Wenxin Yu and Taichu Han and Cheng Yang},
title = {An End-to-End Real-World Camera Imaging Pipeline},
booktitle = {ACM MULTIMEDIA 2024},
year = {2024},
}