Lights’ Clarity Platform – multi-view Depth Perception for the next-generation of Vehicles

Light delivers the most sophisticated depth perception technology in the world, mostly due to their as yet unseen combination of range, detail, accuracy and consistency of depth estimation. Tooploox engineers, researchers, and designers have contributed to the company’s works.

Autonomous vehicles are no longer a buzzword. According to the Markets&Markets report, the value of the global autonomous car market reached $24.1 billion in 2019 and is expected to project a CAGR of 18.06% between 2020-2025. This growth is fueled by both policymakers and companies alike. PwC estimates that up to 40% of mileage driven in 2030 will be done by autonomous vehicles. Also, up to 55% of small business owners believe that within two decades they will have a fully autonomous fleet.

When it comes to autonomous cars, the main media focus is on the artificial intelligence responsible for controlling the car’s behavior on the road (and elsewhere). Yet behind the system which decides whether to accelerate or not, there is a net of sensors which deliver the information required to do so – literally the eyes, ears, and sense of motion of the intelligence behind each decision.

And there can be no correct decision without proper data.

The Client

Light is the company behind the sophisticated multi-lens and multi-sensor cameras initially used in mobile devices. The company’s products have delivered optical zoom, a higher overall image quality, and stereoscopic depth estimation.

Recently, the company has been focused on using its expertise to build perception technology for the next generation of vehicles. Harnessing the experience gathered while delivering smartphone cameras, Light is leveraging its previously-developed multi-view technology to better address the needs of driver-assisted and autonomous vehicles.

What is multi-view and what makes it so special?

On the most basic level, multi-view is based on combining two or more images taken from different angles in order to gain information about its depth and produce a 3D image. The process is widely seen in nature, where images from two eyes are combined in the brain to produce a multidimensional view. Basically the same can be achieved by using the cameras of autonomous cars to make depth estimations.

The process itself is pretty straightforward. When one knows the distance between lenses (either cameras or eyes) and the difference in the position of a particular entity in both images, one can easily calculate depth using the principles of triangulation.

When compared to other depth estimation methods used in autonomous cars, like radar or lidar, multi-view has multiple advantages:

It is passive – using this technique emits no signal (contrary to laser-based lidar or radio-based radar), thus energy consumption stays low and safety remains high.
It delivers data used in multiple systems – the video material gathered by the cameras can be used in image analysis or filtered through existing image recognition solutions without further data collection required.
It works on image data – existing road infrastructure is delivered to drivers through their sight, not radar or lidar. Thus, stereoscopic cameras are the best fit to the current road norms.

But being pretty straightforward on paper doesn’t always carry over into reality. Not even close.

Challenges in Multi-view

One crucial and the most painstaking element of multi-view vision systems is making cameras efficiently work in tandem or in larger groups. There are several challenges to take into account when thinking about this technology:

Camera calibration

every single camera needs to be calibrated and adjusted according to the requirements of the Light Clarity Platform, especially considering the great variability among available models and configurations. On the other hand, this approach allows the company to cut costs significantly by utilizing an already available hardware
Camera movement

it is nearly impossible to eliminate the subtle movement of the camera lens. Even a slight change in position influences the angle between two lenses and can significantly alter estimations
Not knowing the exact position of pixels (only approximate positions)

the key aspect of stereo vision is the ability to find the same entity in two images (be that a tree, a car, a lamp, or a traffic light – it doesn’t matter) and know the difference between its position in images from two or more cameras. If the solution is to work in an autonomous car, it needs to be extremely fast and efficient, it needs to work on incomplete data (the sensor does not load a full image, but strings of data) and remain reliable – a task not to be underestimated
Slightly different pixel world representation

the basis of the computational process behind stereovision is the difference between the positions of two known pixels or a group of pixels that compose what the human eye interprets as a car or a tree. Yet it is not certain that the sensor will interpret the world in the exact same way – the fringe of the object can be blurred or moved due to the nature of the sensor, which changes light into an image. So even assuming the perfect calibration of all cameras, there can still be some misalignment of the rectified images
Different camera photometric calibration

another challenge is in keeping the cameras’ photometric calibrations the same – yet sometimes this is simply impossible. After light is transformed into a stream of bits and processed by machines, matching these can get even more complicated
Perspective distortion

last but not least, when it comes to the cameras used in autonomous cars, a widely known phenomenon of perspective distortion comes into play. The wider the gap between the cameras, the bigger the perspective distortion and the algorithm needs to take this into account

Our solutions

Considering the challenges mentioned above, there is no multi-view depth estimation without a heavy dose of engineering and the solving of multiple challenges, both those known and those unexpected. Thus, the Tooploox engineers had to deliver top-notch solutions from fields such as:

🡒 Signal analysis
🡒 Combination of data from various sources
🡒 Machine learning

The main element of multi-view solutions is signal analysis – the algorithmic processing of the sensor output to produce a real-time depth estimation. Light’s goal is to deliver a product that delivers significantly better performance than the existing state of the art solutions.

To achieve that, several development stages were necessary.

The first step was to build an environment that would enable our engineers and researchers to test various approaches and hypotheses without the need to use hardware which was still in development. In addition, using physical hardware would have delivered multiple risks and obstacles – from complicated data gathering to even more challenging evaluations, so we had even more reason to leave it aside.

Thus, the only reasonable option was to build a simulator. The team delivered two versions of the testing environment. The first was basically a playground for multiple algorithms and approaches with no limitations from the hardware. Once the best approach was selected, the algorithm was tested on a simulated version of the hardware. That’s where the limitations on memory and computing power kick in. This HW emulator was a key factor in many HW design decisions, granting invaluable insights into what would be obtained from the physical chip.

The challenge comes from gathering data. The best option is, of course, to use real data, yet with the hardware stuck in the design phase, there was initially no option to gather data by simply driving around with the device installed.

Also, the data needs to be of the best quality possible, delivering several qualities that are usually hard to obtain – the ground truth depth and exact information about lighting.

Why not use Computer Games?

When it comes to generating data used for autonomous car training, it is popular to use computer games and artificial environments that are similar to them, like Carla. But, this time, the artificial data generated in that way would be not sufficient – computer games use multiple graphics tricks to emulate real images, in fact delivering a convincing gimmick of reality which has been stripped of as much redundant information as possible – delivering real-life graphics in a computer game would require using supercomputers, not a gaming platform like a PC or console.

Thus, the Tooploox team delivered training data that was generated using professional rendering software, in order not to lose any lighting features and retain accurate data about the relations between objects in 3D space.

With this data, the engineering team was able to evaluate several primary concepts that emerged during the design process. Before the real-life data had been gathered, Light engineers, supported by the Tooploox team, managed to deliver a working prototype of the stereoscopic depth estimation algorithm that was ready to be polished with more data and tested in more environments.

Among the key features delivered at this stage of the process was regularization done in real time. While it is fairly easy to do regularization on a static image, doing it with real-time and HW constraints provides additional challenges.

What is Regularization

Regularization is a depth estimation enhancement technique that reduces the risk of mistakes. The algorithm compares depth estimations from nearby fields to ensure that information is consistent in a particular context. Without this, every pixel would be processed fully independently with a higher chance for a variety of mistakes.

With a complete image, regularization is pretty straightforward, but the camera sensor sends data as a constant signal, building an image from “rows” streamlined on a display. The image itself appears complete for humans due to the high refresh rate, yet the machine needs to work on segments of data to do it in real time.

With a good simulator and the data required for validation, Tooploox and Light engineers managed to deliver a regularization algorithm that works on segments, not the full image. The process also enabled the team to significantly reduce the amount of hardware resources required.

While multi-view vision delivers reliable information about the distance between objects, an autonomous vehicle needs as reliable information as possible. And that’s why the Tooploox and Light teams have implemented data-merging features, which use external information to support depth estimations:

GPS – the system uses information from the GPS system to get reliable information about the vehicle’s position.
Accelerometer and gyroscope – the accelerometer delivers information about the vehicle’s movement so adjustments can be made to take the constantly changing position into account.
HD maps – having the exact location of the vehicle and precise information about its motion, the system can use the information stored in HD maps to boost the accuracy of distance estimations and to correct predictions according to data from the map.

What is an HD Map

An HD map is a map enriched with multiple layers of information useful for autonomous vehicles, like the positions of road signs and information about their meaning, the exact positions of buildings and trees and a myriad of other useful bits of data which can be used to improve an autonomous car’s performance.

Using information from the HD map Light system can boost depth estimation accuracy. The information is improved further by comparing the position taken from the camera of a nearby object with its position on an HD map alongside the information from GPS and accelerometer systems.

The system was also augmented with machine learning capabilities that further improve the accuracy of depth estimations. The Tooploox AI team worked on improvements to the state-of-the-art PSMnet (Pyramid Stereo Matching Network) solution in order for it to work in the real-life use case of the Light camera system.

The neural network was designed to serve the purpose of generating depth estimations from the pair of stereoscopic images, basically the main task of Light cameras, and neural networks delivered another layer of validation.

The effect

With support from Tooploox engineers and designers, Light managed to build a unique system that combines multiple advantages as yet unseen on the market:

Top-class accuracy in depth estimations – using only the camera’s input, the system delivers top-class accuracy in estimating depth of up to 1000m away – this range is unmatched.
Energy efficiency – the system uses only passive input, making it significantly more energy efficient than other perception solutions, such as lidar and radar. Also, using only passive input makes the solution more suitable for challenging environments, where signal emission can be disturbed by the presence of sophisticated devices.
Cost efficiency – the platform is delivered using cameras that are already available on the market, significantly cutting the costs of the solution.
Improves performance of the entire sensor stack. – the system not only uses a cutting-edge state-of-the-art depth estimation algorithm, but also validates its output with data taken from other sources, such as GPS, HD maps and neural networks which validate the cameras’ output.
Solid basis for further use cases – the primary use case for Light’s Clarity Platform is to make the next-generation of cars safer, with or without a driver. But that is just the beginning. With this level of depth estimation, there are many more use cases yet to be delivered.

Lessons learned

A testing environment is a must when delivering sophisticated, hardware-embedded software.
With artificial data of reasonable quality, the team can easily test an initial set of hypotheses and filter out the worst-performing iterations to advance their work only with the best, significantly boosting efficiency and reducing delivery time.
On the other hand, there are multiple factors in real data that can deliver unexpected results. Additional research is always necessary when transferring from synthetic to real data usage.

Let our specialists solve the problems and tackle the challenges that hold you from conquering the world.

Let’s talk

Lights’ Clarity Platform – multi-view Depth Perception for the next-generation of Vehicles

The Client

What is multi-view and what makes it so special?

Challenges in Multi-view

Camera calibration

Camera movement

Not knowing the exact position of pixels (only approximate positions)

Slightly different pixel world representation

Different camera photometric calibration

Perspective distortion

Our solutions

🡒 Signal analysis

🡒 Combination of data from various sources

🡒 Machine learning

Delivering a Testing Environment

Artificial Data Production

Why not use Computer Games?

Regularization in Real Time

What is Regularization

Sensor Fusion

What is an HD Map

ML Infusion

The effect

Lessons learned

Let our specialists solve the problems and tackle the challenges that hold you from conquering the world.