Arm Mali-C55: Image processing with smallest silicon area and highest performance – Internet of Things (IoT) blog – Arm Community blogs

The demand for image processing in future devices continues to grow exponentially. One prime example is the security camera and surveillance market, currently worth $25B. It is expected to grow at 13% CAGR to $45B by the year 2025, with roughly a quarter of that in consumer smart home cameras. There are around a billion cameras installed globally. It is estimated that the next billion cameras will be installed in a period of just 4 years. Arm is extending its ISP product portfolio and announcing the latest entrant to the Arm Mali camera family, Arm Mali-C55: the smallest and highest-performance Image Signal Processor (ISP) from Arm to date.

Addressing a wide range of image-processing use cases

Mali-C55 offers the most efficient combination of image quality, high throughput, power consumption, and silicon area. It can enable a broad range of applications including endpoint AI, smart home cameras, AR/VR, and smart displays.

It can meet the expanding needs of vision applications for wired or battery powered devices at different power envelopes in challenging lighting and weather conditions. Mali-C55 can be seamlessly integrated in SoC designs with either Cortex-A or Cortex-M CPUs as it includes industry standard AXI and AHB interfaces. 

Mali-C55 offers a wide range of configurability options, providing the most area-efficient implementation to support targeted use cases. For example, a computer vision-only application would not need the same level of image-processing as a full color display output. With Mali-C55, specific image processing functions can be left out or individually simplified in a modular fashion, resulting in an area optimized silicon footprint.

Mali-C55 also comes with improved image quality due to updates to the color reproduction and noise reduction functions, and has higher performance with up to 1.2Gpix/sec throughput.

Supporting high-quality image output

Building on the class-leading Mali-C52 ISP, the new Mali-C55 image signal processor supports applications requiring high-quality image output by providing enhancements of the following functions:

• Arm Iridix local tone mapping
• Arm Temper temporal noise reduction
• Arm Sinter spatial noise reduction
• High Dynamic Range (HDR) sensors support

Iridix local tone mapping is the process of applying intensity transformations to images to achieve better visualization by using information gathered from local regions within images. Iridix defines these local regions in an image as grids with equal sizes. It extracts statistics from each grid to apply the collected statistics to the corresponding local regions in the image. Compared to Mali-C52, Mali-C55 improves the Iridix local tone mapping algorithm by smoothing each local tone curve therefore enabling a more natural fall-off around bright light sources.

Temper is a temporal noise reduction algorithm that improves the quality of images in low light conditions by combining consecutive frames. Mali-C55 not only improves the image quality with updated noise reduction algorithms but achieves this with up to 50% reduced memory bandwidth compared to Mali-C52.

Sinter 2.6 is an improved spatial noise reduction technique that improves the detail and noise balance in color channels. Compared to Mali-C52, Mali-C55’s Sinter achieves better balance of detail by using specific registers for each color channel.

The Temper and Sinter functional blocks were designed to work together for a significantly better image quality by sharing information between the modules to apply stronger noise reduction in various regions. The Temper and Sinter block order is switched in the pipeline compared with previous ISP designs. This way the input motion mask from Temper improves the overall motion-adaptive noise-reduction performance, while providing per-plane noise profiling.

For computer vision applications where high throughput and low latency is required, the Mali-C55 ISP can be configured by either enabling or disabling the above-mentioned features. This configurability helps deliver the maximum performance and capability in the lowest area, making it ideal for a variety of use cases.

–> Compare Mali family for ISPs

Enhancing ML processing capabilities

Increasingly in modern SoCs, the output of an ISP is directly connected to a machine learning accelerator for further processing via neural networks or other ML algorithms. This usually means providing downscaled images for ML models to detect and identify objects, or even attempting more complex inferences such as pose estimation. Mali-C55 provides a second output pipe that can output downscaled images suitable for input to any machine learning accelerator. Pairing an ISP with a machine learning accelerator adds ML capabilities to the image-processing pipeline by being able to use other transformer models as denoising techniques, which further enhances image quality.

Because of the ever-increasing demand for higher resolutions, memory bandwidth is expected to be a key bottleneck in upcoming vision orientated SoCs. Adding a machine learning accelerator to such an SoC can further increase the bandwidth requirements, thus exacerbating the problem. Mali-C55 enables integration with other products such as Arm Frame Buffer Compression (AFBC), that can address such bandwidth issue and provide great benefit at the system level.

Mali-C55 can seamlessly integrate with complementary products including:

• Machine learning accelerators
• Arm Frame Buffer Compression (AFBC)
• Arm Mali-C10 Geometric Distortion Correction (GDC)

This combination of technologies can achieve an optimal imaging solution at a system-wide level.

ML-enabled cameras are enabling a new paradigm – the “Software Defined Cameras” – which provides new ways for OEMs and service providers to deploy cameras around the world. Key hardware capabilities are provided up-front, but with sufficient ML and general-purpose processing to improve or enable additional software features as time progresses. Subscription and other revenue models are being developed around these dynamic feature capabilities. We expect Mali-C55 ISP to be a key component for the flexible and capable compute systems contributing to this trend.

Improved multiple image processing

With multi-camera processing for up to 8 inputs, Mali-C55 can support up to 48Mpixel / 8K resolution. The previous generation, Mali C-52 only supports up to 4 cameras through a single interface and a maximum of 16 Mpixel. Mali-C55, with the best combination of image quality, throughput, power consumption, and silicon area, meets the requirements of a wide range of consumer, commercial and industrial vision system applications.

Software and tools support

As with any Arm solution, software and tools comprise an important part of the Mali-C55 offering. A complete software package is available for Mali-C55 licensees for controlling the following functions:

• ISP
• Sensor
• Auto White Balance, Auto-Focus, and Auto Exposure
• Bare-metal and Linux (Video4Linux framework – V4L2) support

ISP users also need the capability to tune both objectively and subjectively, and Arm provides a full set of tuning and calibration tools. Arm tools help developers to achieve the desired image quality, which is paramount for vision devices. Arm software and tooling enable optimal designs and reduce time to market.

Arm also offers specialized tuning training courses and support for tailored tuning to meet specific sensors and use cases. 

To speed up of SoC development with Mali-C55 before silicon availability, Arm also offers a bit-exact simulation model along with a reference platform, which enables the pre-built and pre-tuned evaluation of Mali-C55 image quality. 

Arm continues to enable more partners in the ecosystem to provide ISP tuning services for all Mali ISP products including Mali-C55.

Delivering on the Arm Total Solutions for IoT roadmap

Over the last year, Arm has unveiled Arm Total Solutions for IoT to accelerate IoT development. As part of the roadmap, we plan to include Mali-C55 in upcoming Total Solutions for IoT – starting with a full reference design for vision systems. The Mali-C55 image signal processor will come with a pre-validated solution that will support a specific sensor and dual output mode which seamlessly connect to a machine learning accelerator that performs various functions.

Summary

The Mali-C55 ISP is the next step in the evolution of Arm’s imaging products. It will enable our partners to offer market leading products in the ever-expanding smart camera space, with significantly upgraded performance and efficiency compared to existing IoT camera systems. 

Find out more about the capabilities of the Mali-C55 image signal processor and reach out to Arm for support with your next vision-based SoC design.

Learn More about Mali-C55