Bid Farewell to ISP Tuning and Manual Parameter Tweaking with RZ/V2M “Tuned ISP“

Recently, the majority of camera sensor input has become 2Mpix (full-HD) or 8Mpix (4K). However, when using high-resolution sensors, it is common problem that there are few choices of sensor modules that support ISP functionality. Also, the module with ISP is more expensive than the module without ISP. Therefore, the MPUs that support ISP is more desirable to manipulate image data and improve image quality.

This use case does not require an external ISP chip, thus you can reduce the BOM cost of your product. On the other hand, ISP tuning is a time-consuming task because of manual parameters tuning and effectively it adds schedule delays and additional development.

Consider some of the challenges listed below:

• Multi-function implementation challenges
  Typically, ISP converts image data and simultaneously adds image improvements:
  e.g., WDR that blends two or more images to enhance the image, 2D or 3D noise reduction

• Complexity challenges
  The number of registers also increases with increased ISP functionality.
  In some cases, there were thousands of registers in the ISP.
  You must adjust these registers to get the desired image quality.

• Sensor-specific performance and availability challenges
  The best setting depends on the sensor type because these settings need to consider the unique characteristics of the sensor such as color.
  There are also differences in settings depending on the lens, for example, shading and distortion correction.

Even for users who are familiar with ISPs, many challenges need to be overcome and iterations are needed. Therefore, at RENESAS, we came up with a "Tuned ISP" concept to simplify the ISP functions of the RZ/V2M MPU to speed up the image improvement process and shorten development time. Renesas uses APIs to abstract each ISP function at a higher level without compromising the image quality of the ISP function.

The "Tuned ISP" has the following features:

• Various ISP functions are supported as API.
  The several functions such as auto exposure or auto white balance require additional software processing.
  The API includes the processing, thus you can realize the total function only by calling the API.

• The image quality by default starts at a baseline that is optimum.
  From this setting, you can apply the fine adjustments using the parameters of API.

• You can use the adjustment tool when you install your product in the field.
  This tool also supports many functions such as exposure and distortion correction.
  

The setting by this tool is referred to by API, but you can change the setting without much parameter tuning or programming.

You can "easily" and "immediately" use the input images from the sensor with preset settings and APIs.This figure shows the support function of the RZ/V2M ISP.

Renesas also provides multimedia-related APIs. These can be easily implemented on encoded video streams after ISP conversion.

To get started, Renesas has selected the number of sensors that are supported on RZ/V2M and the APIs Renesas provides. We continue to expand the list of supported sensors by RZ/V2M’s "Tuned ISP".

Please refer to the functionality of API and related software.

1. RZ/V2M: The product with high-performance ISP and power efficiency AI inference

2. RZ/V2M ISP Support Package: Supports ISP and multimedia API 

3. Introduction for Image Quality Functions listed the several API results

4. RZ/V2M Linux Package: Base software for RZ/V2M

5. RZ/V2M DRP-AI Support Package: supports AI inference function on RZ/V2M.