Tooploox, alongside researchers from Wroclaw University of Science and Technology, has pushed the boundaries of state-of-the-art methods to improve the quality of images with low light conditions by introducing the Dimma system.
Today, most users have a smartphone that offers image quality only available on professional photographic devices just a decade ago. Moreover – the visually breathtaking video for Lady Gaga’s “Stupid Love” was shot only using an iPhone 11, with no pro cameras involved.
Yet the relationship between cameras and lighting started off as complicated, to say the least. In 1826, Nicephore Niepce arguably captured the first photograph, yet it required a full eight hours of exposure – nothing to be practically useful for any real purpose. The first method of actually making photography public – the Daguerreotype, was invented by Louis Daguerre in 1833, gaining popularity in the 1840s and 1850s.
One of the drawbacks was the need for a long period of light exposure – which meant about 15 minutes of sitting completely still in near-blinding light, which was necessary for the chemical reagents to act. This is why the first reliably dated image of a human being was made by accident – Louis Daguerre himself was taking a long exposure picture of a busy street, unlikely and unintended to capture anyone, yet there they were. A solitary gentleman and a shoe polisher attending to him.
The need for bright light and to sit still for a long time are also the reasons behind the grim faces of people photographed throughout the 19th century.
Painting with light
These days, cameras are capable of capturing images of moving objects like cars and planes, wild animals as they leap mid-air, or playing with children – assuming the light conditions are good enough. If not – the photograph is dark, blurred, unclear, and noisy (in a data-related meaning.)
Smartphone and camera manufacturers are doing their best to address this challenge by improving both hardware and software. Last but not least, artificial intelligence can be used to adjust low-light photography settings in real time or improve the quality of images after they are taken.
The Tooploox research team followed the latter path, challenging industry standards and state-of-the-art techniques.
Challenges
Improving the quality of images that suffer from light underexposure is not only a practical challenge to solve but also an interesting scientific problem to tackle. However, there are at least three different aspects one needs to consider when working on this matter.
Models can vary in performance depending on the camera used
There are thousands of camera models that vary in terms of their lenses, their Charge Coupled Devices, their software, and a multitude of other factors – starting from super-sensitive professional devices to ones used in toys or nearly-toylike devices. Every combination of these factors creates a unique type of distortion in dim light that may be illegible for a model trained on a dataset that was not diverse enough.
Building the dataset
The internet and public domain are full of high-quality images of nearly everything. The challenge is to find images that suffer from light underexposure – people are not that eager to share photos that are of bad quality or technically lacking.
Also, the core of the dataset would ideally be built of a pair of photos depicting the same scene, with one being of top quality and the other being a dimly lighted one. The likelihood of the existence of such pairs is extremely low. Also, the dataset should include these pairs from a variety of cameras, making the model less device-dependent. And last but not least – it must contain multiple scenes, from smiling people to landscapes, from cars to… you name it.
So, as you can see, such a dataset is extremely challenging to build.
Our approach
Having the challenges in building the dataset on hand, one needs a smart approach to tackle the problem without investing too much. The machine learning skills of the Tooploox Research team are once again the tool to overcome this. To do so, the team took the following steps:
- Create the starting point datasets – the team prepared photos from LOL – a publicly available machine learning dataset. To enhance data availability and quality, some pairs had to be produced manually, using the available tools and cameras – basically, some members of the team took pairs of orchestrated photos to include in the dataset.
- Extend the dataset using ML – the team utilized the semi-supervised technique to extend the set of labeled examples using the dimming machine-learning approach, creating a darker corresponding pair for existing, well-lit photos.
- Build the camera-mimicking tool – last but not least, various cameras and devices approach dim light in different ways, generating noise and blurring images in a highly context-dependent manner. To tackle the challenge, the team came up with a Mixture Density Network that distorts the colors in an image in the way a particular camera would, further enhancing the quality of the artificial data.
Shedding some light
Having prepared the dataset, delivering the solution became much easier. In some way, it was about reversing the process described above – the task of the model was to recreate perfect and undistorted images from dimmed and altered ones.
The Tooploox Research team has devised a Diffusion models-based solution that applies the desired level of brightness as well as adjusts the colors to fit the desired quality. Experiments have shown that the solution achieved results comparable to current state-of-the-art supervised methods and outperformed all unsupervised methods. Also, the solution achieved state-of-the-art results in all applicable tests.
Limitations
The key drawback of the Tooploox Research-delivered solution is the necessity to train the solution using a dimming module calibrated for particular camera models and settings. Making the model more general in the future may be a solution.
Advantages and potential use cases
Using the data generating module makes the solution cheaper and faster to train than any of the leading technologies used now.
This tool can be used in commercial applications in photography and filmmaking, clearing and correcting the light levels of images and movies. The tool can also be used to restore an image that would otherwise be lost due to insufficient light exposure.
Also, the technique can prove helpful when the need to extract information from an image arises – for example, for scientific or investigative purposes.
The paper detailing the research can be found on Arxiv.
