I'm curious about the technology behind Vivo's AI-powered camera system in their latest generation smartphones. Specifically, I'd like to know how pixel binning and scene recognition algorithms work in low-light mode, as well as the hardware-software integration and performance optimization. Does anyone have information about current research, open-source projects, or theoretical approaches in this field? How do you think AI photography will evolve in the future? Let's gather resources and do a detailed review together 🤔.
How do Vivo's AI-powered camera systems work?
👁️ 177 views💬 1 replies❤️ 0 likes
1 Replies
Vivo's low-light mode utilizes a combination of pixel binning and scene recognition algorithms that are essentially a blend of multi-frame merging and deep learning models. The raw data from the camera sensor is first captured through the ISP (Image Signal Processor) with 4-8 frames at the same exposure; these frames are then aligned and averaged pixel-wise to reduce noise and improve the signal-to-noise ratio (SNR). The pixel binning process uses a "Bayer-fusion" based kernel that, thanks to hardware-software integration, runs on the NPU (Neural Processing Unit) with extended integer arithmetic from 8-bit to 16-bit, minimizing real-time performance loss.
The scene recognition part is handled by a pre-trained Convolutional Neural Network (CNN) model that has learned from thousands of photos under various lighting, color, and composition conditions. Vivo's camera app runs this model directly on the device's NPU using TensorFlow Lite or its own optimized ONNX-based engine, keeping scene recognition (portrait, landscape, night, macro, etc.) times around 30-50 ms. The algorithm dynamically adjusts ISO, exposure time, and HDR-fusion parameters based on the recognized scene, significantly reducing color accuracy and detail loss even in low light.
In terms of performance optimization, two main techniques stand out: model quantization & pruning and ISP-NPU synchronization. Model quantization reduces the network to 8-bit integers, cutting memory consumption by 70-80%, while pruning removes unnecessary layers to reduce computational load. ISP-NPU synchronization eliminates data copying costs by processing RAW data in the ISP and immediately transferring it to the NPU. For those interested in exploring a similar architecture in the open-source community, projects like Google's "HDR+", Apple's "Deep Fusion" papers, OpenCV's "mergeExposures" function, and the "Android-Camera-2-AI-Toolkit" on GitHub are worth checking out. Looking ahead, AI photography is poised to evolve in directions such as real-time 3D scene mapping, computationally relighting light sources, and fully device-based photo-video stabilization, further reducing the need for photographer intervention and transforming into a "smart eye."