How do Vivo's new camera features work and what's their development potential?

25 Tem 08:44

Vivo’s latest models feature modes like “AI-Scene Detection,” “Portrait Lighting,” and “Night Mode,” which operate through a three-stage image processing pipeline: **pre-processing → model-based prediction → post-processing**. In the first stage, the RAW image data undergoes basic adjustments such as sensor optimization (noise reduction, dynamic range expansion), and color correction. While this raw data passes through hardware-accelerated blocks like Qualcomm’s Snapdragon ISP or MediaTek’s ISP, it is simultaneously fed into a low-power NPU (Neural Processing Unit), where trained convolutional neural networks (CNNs) perform scene classification and object recognition. For example, when a landscape photo is detected, the “Landscape” classification is triggered, activating modules like color-tone adjustments, HDR-fusion, and sky enhancement. The **Portrait Lighting** algorithm used in portrait mode generates a two-stage depth map. First, a depth map is created using stereo-parallax or time-of-flight (ToF) sensors like LiDAR. Then, a multi-scale U-Net architecture refines this map to separate the foreground (face) from the background. Based on this map, the direction of the light source is estimated, and artificial “Rembrandt” or “studio” lighting effects are applied via **render-time shading**, adjusting shadows and highlights accordingly. For Night Mode, multiple low-exposure (long-exposure) frames are merged. Here, the NPU executes a GAN-based process known as **Burst Denoising** and **Computational Night Enhancer**. This network restores details lost in low light while minimizing color distortion. Finally, ISP-based adjustments are made for contrast and color balance. In terms of hardware integration, Vivo’s AI Camera chip (e.g., the X-Series NPU) runs a **quantized** (8-bit) model compiled with mobile-optimized frameworks like **TensorFlow Lite** and **Caffe2**, enabling real-time predictions in milliseconds. Model updates are delivered via OTA (Over-The-Air) updates, adding new scene classes and lighting scenarios regularly. This architecture not only reduces power consumption but also delivers results **30-40% faster** than GPU-based solutions in terms of image quality. If you're interested, the Google AI blog post titled **“Mobile AI Camera Architecture: From ISP to NPU”** and Qualcomm’s **“Snapdragon Camera 2.0”** whitepaper are great starting points. Additionally, the **OpenCV-AI-Pipeline** open-source project on GitHub lets you examine a similar image processing pipeline. By exploring these resources, you can replicate Vivo’s AI-based camera modes in your own test environment and perhaps even develop new model suggestions.

25 Tem 09:40

Vivo’s AI-assisted camera modes typically operate through a three-stage pipeline: pre-processing, scene recognition/portrait lighting, and post-processing. During pre-processing, the RAW data is converted from 12-bit or 14-bit to 8-bit, using a mobile-optimized CNN (such as a MobileNet-V2-based model) for HDR fusion and noise reduction. Before moving to the scene recognition step, this model quickly analyzes the photo’s brightness, color, and contrast to classify it as “night,” “portrait,” or “landscape,” among others. Portrait lighting and night mode are supported by two different micro-models. In portrait lighting, a depth-estimation network extracts a face-specific depth map, blurs the background, and adds a virtual spotlight effect to the light source. In night mode, the long-exposure photo is divided into multiple partial frames, each processed sequentially by low-light noise reduction (denoising) and super-resolution (similar to ESRGAN) networks before being merged. Most of these networks run on TensorFlow Lite or Qualcomm Snapdragon’s Hexagon DSP to minimize CPU load. In terms of hardware integration, these algorithms are tightly synchronized with the Snapdragon 8-Gen 2’s 4-camera ISP and Neural Processing Engine (NPE). The ISP directly transmits RAW data to the NPE, which can complete tasks like face recognition, scene recognition, and low-light enhancement in just 10-15 ms—enabling a real-time preview experience. Now, how much space do these models take up on the device? If the model size were increased by 30%, would photo capture latency increase, or could DSP optimizations keep it just as fast? At this point, it’s important to consider model compression techniques like pruning and quantization.